CN115203217A

CN115203217A - Data synchronization method, device, equipment and computer readable storage medium

Info

Publication number: CN115203217A
Application number: CN202210699262.6A
Authority: CN
Inventors: 刘红云
Original assignee: Ping An Bank Co Ltd
Current assignee: Ping An Bank Co Ltd
Priority date: 2022-06-20
Filing date: 2022-06-20
Publication date: 2022-10-18

Abstract

The embodiment of the application provides a data synchronization method, a device, equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring an SQL statement received by a main database; writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule; and updating the target slave database corresponding to the target log according to the SQL statements written in the target log to obtain the synchronized target slave database. According to the method provided by the embodiment of the application, only the logs and the screening rules corresponding to the slave databases need to be configured in advance, after the SQL sentences received by the master database are obtained, the SQL sentences can be written into the corresponding target logs according to the screening rules, and then the corresponding slave databases can be updated respectively according to the SQL sentences written in the logs.

Description

Data synchronization method, device, equipment and computer readable storage medium

Technical Field

The embodiment of the application relates to the technical field of database synchronization, in particular to a data synchronization method, a data synchronization device, data synchronization equipment and a computer readable storage medium.

Background

In most scenarios, the slave databases of the system sub-modules all depend on the data in the system master database, and the maintenance of the synchronism of the data is a precondition for ensuring the normal and stable operation of the system.

However, the data relied on by each sub-module may be different, and the system master database needs to query the corresponding data according to the property of the slave database and send the data to each sub-module respectively, so as to complete the synchronization between the slave database and the master database. However, this process is very inconvenient, inefficient in synchronization, and prone to synchronization anomalies.

Disclosure of Invention

The embodiment of the application provides a data synchronization method, a data synchronization device, data synchronization equipment and a computer readable storage medium, and aims to solve the problems that an existing data synchronization method is low in synchronization efficiency and easy to generate abnormal synchronization.

In one aspect, an embodiment of the present application provides a data synchronization method, including:

acquiring an SQL statement received by a main database;

writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule;

updating a target slave database corresponding to the target log according to the SQL statements written in the target log to obtain a synchronized target slave database;

writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule, wherein the step of writing the SQL statement into the target log corresponding to the SQL statement comprises the following steps:

respectively screening the SQL sentences according to screening rules corresponding to the preset candidate logs to obtain screening results of the preset candidate logs;

and taking the candidate log with the screening result of passing in the preset candidate logs as a target log corresponding to the SQL statement, and writing the SQL statement into the target log.

As a feasible embodiment of the present application, before the SQL statements are respectively filtered according to the filtering rules corresponding to the preset candidate logs and the filtering results of the preset candidate logs are obtained, the method includes:

creating an initial log and displaying a preset configuration interface of the initial log;

determining a target screening rule according to rule setting operation on the preset configuration interface;

and setting the target screening rule as the screening rule corresponding to the initial log, and taking the set initial log as a candidate log.

As a possible embodiment of the present application, the updating, according to the SQL statement written in the target log, the target slave database corresponding to the target log includes:

if a data synchronization request is received, or the number of SQL statements written in the target log is greater than a preset threshold value, determining a target slave database corresponding to the target log according to a preset database identifier corresponding to the target log;

and updating the target slave database according to the SQL statements written in the target log.

As a possible embodiment of the present application, before writing the SQL statement into the target log corresponding to the SQL statement according to a preset filtering rule, the method further includes:

analyzing the SQL statement to obtain the instruction type of the SQL statement;

and if the instruction type is a data manipulation language or a data definition language, executing the step of writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule.

As a possible embodiment of the present application, after the target corresponding to the target log is updated from the database according to the SQL statement written in the target log, the method further includes:

and deleting the SQL statements written in the target log.

As a feasible embodiment of the present application, after the target slave database corresponding to the target log is updated according to the SQL statement written in the target log to obtain a synchronized target slave database, the method further includes:

extracting associated data from the main database according to a screening rule corresponding to the target log;

determining first summary information of the associated data and second summary information of the synchronized target slave database data;

and if the first abstract information is different from the second abstract information, updating the synchronized slave database according to the associated data.

On the other hand, an embodiment of the present application further provides a data synchronization apparatus, including:

the acquisition module is used for acquiring the SQL sentences received by the main database;

the determining module is used for writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule;

the synchronization module is used for updating the target slave database corresponding to the target log according to the SQL statements written in the target log to obtain a synchronized target slave database;

and taking the candidate log with the screening result of passing in the preset candidate log as a target log corresponding to the SQL statement, and writing the SQL statement into the target log.

On the other hand, the embodiment of the present application further provides a data synchronization device, where the data synchronization device includes a processor, a memory, and a data synchronization program stored in the memory and executable on the processor, and the processor executes the data synchronization program to implement the steps in the data synchronization method.

On the other hand, the embodiment of the present application further provides a computer-readable storage medium, where a data synchronization program is stored on the computer-readable storage medium, and the data synchronization program is executed by a processor to implement the steps in the data synchronization method.

According to the embodiment of the application, for each sub-module in the system, only the corresponding log and the screening rule thereof need to be configured in advance, after the SQL statement received by the main database is obtained, the SQL statement is written into the corresponding target log according to the set screening rule, and the corresponding slave databases can be updated subsequently according to the SQL statement written in each log, so that the updating of the slave databases corresponding to different sub-modules is completed respectively, the whole process does not need to depend on data screening of the main database, the synchronization efficiency is improved, and the synchronization abnormity is avoided.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an implementation scenario of a data synchronization method provided in an embodiment of the present application;

FIG. 2 is a flow chart illustrating steps of a data synchronization method provided in an embodiment of the present application;

FIG. 3 is a flowchart illustrating steps of screening and writing SQL statements into a log according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating steps of generating a candidate log provided in an embodiment of the present application;

FIG. 5 is a flow chart illustrating a procedure for updating a slave database provided in an embodiment of the present application;

FIG. 6 is a flowchart illustrating a procedure of parsing a sentence to determine whether to write a log according to an embodiment of the present application;

FIG. 7 is a flowchart illustrating a step of verifying a synchronization result provided in an embodiment of the present application;

fig. 8 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any inventive step, are within the scope of the present invention.

In the embodiments of the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the invention. In the following description, details are set forth for the purpose of explanation. It will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and processes are not shown in detail to avoid obscuring the description of the invention with unnecessary detail. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in the embodiments herein.

Embodiments of the present application provide a data synchronization method, an apparatus, a device, and a computer-readable storage medium, which are described in detail below.

In the embodiment of the application, the data synchronization method is deployed on the data synchronization device in the form of a program, the data synchronization device is installed in the data synchronization device in the form of a processor, and the data synchronization device in the data synchronization device executes the program corresponding to the data synchronization method to execute the following steps: acquiring an SQL statement received by a main database; writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule; and updating the target slave database corresponding to the target log according to the SQL statements written in the target log to obtain a synchronized target slave database, thereby completing the synchronization of the slave database and the master database.

As shown in fig. 1, fig. 1 is a schematic view of an implementation scenario of a data synchronization method provided in an embodiment of the present application, and the implementation scenario provided in the embodiment of the present application includes a master database 110, a data synchronization apparatus 120, and a plurality of slave databases 130, where the master database may be understood as a database of a system, which includes all data required in a system operation process, and the slave databases 130 may be understood as databases of sub-modules in the system, which include data required in the sub-modules during operation, that is, partial data in the master database, and the data synchronization apparatus 120 mainly implements synchronization between data in the slave databases and corresponding data in the master database by operating the data synchronization method provided in the present application.

Specifically, the data synchronization device 120 may be understood as a plug-in of the system, and is composed of an interceptor 121 and a plurality of logs 122, after the data synchronization device 120 intercepts the SQL statements written in the main data 110 by the interceptor 121, the SQL statements are written in the corresponding logs by using the screening rules corresponding to the logs 122, and then when the synchronization requirement is met, the corresponding slave database is processed according to the SQL statements written in the logs 122, so that the synchronization between the data in the slave database and the corresponding data in the main database is realized.

It should be noted that the implementation scenario diagram of the data synchronization method shown in fig. 1 is only an example, and the implementation scenario diagram described in the embodiment of the present application is for more clearly explaining the technical solution of the embodiment of the present application, and does not constitute a limitation to the technical solution provided by the embodiment of the present application.

Based on the implementation scenario diagram of the data synchronization method, a specific embodiment of the data synchronization method is provided.

As shown in fig. 2, fig. 2 is a schematic flow chart illustrating steps of a data synchronization method provided in an embodiment of the present application, where the data synchronization method in the embodiment of the present application includes steps 201 to 203:

201, acquiring SQL statements received by a main database.

For the description of the main data, reference may be made to the implementation scenario diagram provided in fig. 1, and details of the embodiment of the present application are not repeated herein. There are many ways for the data synchronization device to obtain the SQL statement data received by the master database, for example, the SQL statement received by the master database is intercepted by setting an interceptor, and certainly, the data synchronization device may also be disguised as a slave database by simulating a slave interaction protocol of mysql to obtain a binary log of the master database, so as to further extract the SQL statement received by the master database. However, the SQL statements received by the main database acquired by the interceptor can be realized in a plug-in mode without intruding into the system, so that the system is little in modification amount, and the operation of the system is not influenced. Therefore, as an optional embodiment of the present application, the data synchronization apparatus employs an interceptor to intercept the SQL statement received by the master database. Specifically, a mycat interceptor or other self-defined feasible interceptor may be adopted, and taking mycat interception as an example, the program statements for intercepting the SQL statements are as follows:

an interceptor: iocat server interrupt impl statics sqlIntercaptor

The interception type is as follows: UPDATE \ DELETE \ INSERT.

202, writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule.

In the embodiment of the present application, the data synchronization apparatus includes a plurality of logs and a preset screening rule, where the logs may exist in a form of binary logs, and details of the embodiment of the present application are not repeated herein. Specifically, each log has a preset screening rule, at this time, the data synchronization device processes the SQL statement according to the screening rule of each log, determines whether the SQL statement satisfies the screening rule of each log, and writes the SQL statement into all the logs satisfying the screening rule, and the specific implementation scheme may refer to the content of fig. 3 and its explanation.

Further, as an optional embodiment of the present application, it is considered that data synchronization is mainly directed to synchronization of data that has been updated or changed in a main database, and therefore, for SQL statements that do not change data in a database partially, such as query statements, do not need to be written into a log, and therefore, after the SQL statements in the main database are captured, the SQL statements are parsed to determine the type of the SQL statements, and whether the SQL statements need to be written into the log is determined according to the type of the SQL statements. The specific implementation scheme can refer to the content of the subsequent fig. 6 and the explanation thereof.

And 203, updating the target slave database corresponding to the target log according to the SQL statements written in the target log to obtain a synchronized target slave database.

In the embodiment of the application, after the SQL statement is written into the target log corresponding to the SQL statement, the target slave database corresponding to the target log is updated according to the written SQL statement in the target log, so as to obtain the synchronized target slave database.

It should be noted that, instead of updating the slave databases corresponding to the respective logs based on the SQL statements written in the respective logs each time the SQL statements are written in the logs, the data synchronization device updates the slave databases corresponding to the respective logs based on the SQL statements written in the respective logs after a set synchronization condition is satisfied. That is to say, before the set synchronization condition is not satisfied, the data synchronization device continues to continuously acquire the SQL statements received by the master database in the form of an interceptor, and writes the acquired SQL statements into the corresponding logs according to the preset screening rule, and after the synchronization condition is satisfied, updates the slave databases corresponding to the logs respectively by the SQL statements written in the logs, thereby implementing synchronization of the slave databases. The following fig. 5 and its explanation can be referred to for a specific implementation.

Further, as another optional embodiment of the present application, after the SQL statements written in the target log are used to update the target slave database corresponding to the target log and obtain the synchronized target slave database, the SQL statements written in the target log may be deleted, so as to avoid repeatedly using the same SQL statements to update the target slave database, and at the same time, the storage amount of the log is also saved, and the storage resource of the data synchronization apparatus is saved.

Further, as another optional embodiment of the present application, after the target slave database corresponding to the target log is updated by using the SQL statement written in the target log to obtain the synchronized target slave database, the data in the master database is further used to check to determine whether there is an abnormal update problem, and for a specific implementation scheme of checking by using the data in the master database, reference may be made to the following fig. 7 and the contents explained in the following fig. 7.

As shown in fig. 3, fig. 3 is a schematic flow chart illustrating a step of screening and writing SQL statements into a log according to an embodiment of the present application, which is described in detail below.

In the embodiment of the present application, the data synchronization apparatus may respectively perform screening processing on the SQL statements according to screening rules corresponding to candidate logs preset therein, specifically, the method includes steps 301 to 302:

301, respectively performing screening processing on the SQL statements according to the screening rules corresponding to the preset candidate logs to obtain the screening results of the preset candidate logs.

In the embodiment of the application, the preset candidate logs in the data synchronization device are respectively configured with corresponding screening rules, and the screening result of whether the SQL statement meets each preset candidate log can be obtained by screening the SQL statement based on the screening rules. Specifically, the screening result includes pass and fail, by which is meant that the slave database corresponding to the log includes a data table corresponding to the SQL statement, and the SQL statement needs to be written into the log, so as to facilitate subsequent synchronization of the slave database. Otherwise, failing to pass means that the slave database corresponding to the log does not include the data table corresponding to the SQL statement.

Further, the screening rule corresponding to each preset candidate log may be configured in a database table manner, an xml configuration file, or a configuration center (e.g., user-defined namespace storage on apollo) manner, and the like, and this embodiment of the present application is not described herein again. Specifically, the specific implementation scheme of pre-configuring the logs corresponding to the slave databases and the screening rules thereof can refer to the following fig. 4 and the explanation thereof.

302, taking the candidate log with the screening result of passing in the preset candidate log as a target log corresponding to the SQL statement, and writing the SQL statement into the target log.

In the embodiment of the present application, it can be known by combining the foregoing description that the candidate log whose screening result is a pass means that the slave database corresponding to the log includes a data table corresponding to the SQL statement, and therefore, it is necessary to consider that the log is used as a target log corresponding to the SQL statement, and write the SQL statement into the target log. Specifically, there may be a plurality of target logs, but the implementation of the technical solution of the present application is not affected.

According to the method and the device, the SQL sentences are respectively screened by using the screening rules corresponding to the preset candidate logs, the screening results of the preset candidate logs are obtained, so that the logs related to the SQL sentences are determined from the preset candidate logs as the target logs, the SQL sentences are written into the corresponding target logs, and the target logs can be used for updating the corresponding sub-database subsequently.

As shown in fig. 4, fig. 4 is a flowchart illustrating a step of generating a candidate log according to an embodiment of the present application, which is described in detail below.

In the embodiment of the present application, an implementation scheme for generating a candidate log based on user configuration is provided, and specifically, the implementation scheme includes steps 401 to 403:

401, creating an initial log, and displaying a preset configuration interface of the initial log.

In the embodiment of the application, when the data synchronization device accesses the sub-module of the system, that is, the slave database, the data synchronization device creates an initial log corresponding to the sub-module, and simultaneously displays a preset configuration interface of the initial log, where the preset configuration interface includes a plurality of rules.

402, determining a target screening rule according to the rule setting operation on the preset configuration interface.

In the embodiment of the application, a user can select and combine the rules on the preset configuration interface, and the final target screening rule can be obtained based on the rule setting operation on the preset configuration interface.

And 403, setting the target screening rule as the screening rule corresponding to the initial log, and taking the set initial log as a candidate log.

In the embodiment of the application, the target screening rule generated by the rule setting operation on the preset configuration interface is the screening rule corresponding to the initial log, further, the initial log at the moment is the candidate log in the data synchronization device, and when the corresponding SQL statement is written in, the slave database corresponding to the log can be updated.

Further, as can be seen from the foregoing description, when the initial log is created, the initial log is created based on the slave database accessed by the data synchronization device, and therefore, the created initial log and the accessed slave database are associated with each other, and therefore, the database identifier of the slave database can be further associated with the initial log to obtain a candidate log including the filtering rule and the database identifier, so that the writing of the SQL statement is completed based on the filtering rule subsequently, and the synchronization corresponding to the slave database is completed based on the database identifier.

Fig. 5 is a flowchart illustrating steps for updating the slave database according to an embodiment of the present application, and is described in detail as follows.

In the embodiment of the present application, an implementation scheme for updating a database based on a specific synchronization condition is provided, which specifically includes steps 501 to 503:

501, judging whether a data synchronization request is received or not, or judging whether the number of the SQL statements written in the target log is greater than a preset threshold value or not. If yes, go to step 502; if not, other steps are executed.

In the embodiment of the application, when the target slave database needs to perform data synchronization, a data synchronization request is sent to the data synchronization device, so that after the data synchronization request sent from the database is received, the target slave database can be updated by using the SQL statements written in the log, so as to implement data synchronization. In addition, when the number of the SQL statements written in the target log is greater than the preset number threshold, the target is also updated from the database by using the SQL statements written in the log, so as to empty the SQL statements written in the log, and save the storage space.

502, determining a target slave database corresponding to the target log according to a preset database identifier corresponding to the target log.

In the embodiment of the present application, specifically, as can be seen by combining the foregoing description, in the process of configuring the logs, the database identifiers corresponding to the logs are configured synchronously, and therefore, when the data synchronization condition is satisfied, the target slave database corresponding to the target log is determined according to the preset database identifier corresponding to the target log, so as to complete subsequent data synchronization.

503, updating the target slave database according to the SQL statements written in the target log.

In the embodiment of the present application, the SQL statement written in the target log is used to process the data in the target slave database, so as to obtain the updated database, and the specific process of executing the SQL statement is not described herein again in this embodiment of the present application.

As shown in fig. 6, fig. 6 is a flowchart illustrating a step of parsing a sentence to determine whether to write a log according to an embodiment of the present invention, which is described in detail below.

In the embodiment of the present application, a technical solution is provided for determining a command type by analyzing an SQL statement, so as to determine whether to write a log, and specifically includes steps 601 to 603:

601, analyzing the SQL statement to obtain the instruction type of the SQL statement.

In the embodiment of the present application, the instruction type of the SQL statement generally includes three types, one of which is a data definition statement and is used to create or delete objects such as a database for storing data and a table in the database. Specifically, the data definition statement includes the following instructions: CREATE (CREATE database and table objects), DROP (delete database and table objects), and ALTER (modify the structure of database and table objects), among others. The second is a data manipulation language used for inquiring or changing the records in the table, and the data manipulation language comprises the following instructions: INSERT (INSERT new data into the table), UPDATE (UPDATE data in the table), DELETE (DELETE data in the table), and so on. And the third is a data control language used for confirming or canceling the change of the data in the database, and the data control language comprises the following instructions: COMMIT (confirming changes made to data in the database), ROLLBACK (canceling changes made to data in the database), GRANT (granting user operating rights), and REVOKE (canceling user operating rights), among others.

And 602, judging whether the instruction type is a data manipulation language or a data definition language. If yes, go to step 603; if not, other steps are executed.

In the embodiment of the application, if the instruction type is a data manipulation language or a data definition language, it is considered that the SQL statement will update data or a data structure in the master database, and may involve synchronization of data in the slave database, and therefore, the SQL statement needs to be written into a corresponding log, whereas if the instruction type is a data control language, it is considered that the SQL statement does not update data or a data structure in the master database, and therefore, the SQL statement does not need to be recorded in the log.

603, writing the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule.

In the implementation of the application, the command type of the SQL statement is obtained by analyzing the SQL statement, whether the SQL statement adjusts the data or the data structure in the main data is judged by using the command type, and the SQL statement which may adjust the data or the data structure in the main data is written into the log, so that the efficiency of data synchronization can be further improved.

As shown in fig. 7, fig. 7 is a schematic flowchart of a step of verifying a synchronization result according to an embodiment of the present application, which is described in detail below.

In the embodiment of the present application, after synchronizing the slave database by using the SQL statements written in the log, the data in the master database is also used for checking to determine whether there is an abnormality, which specifically includes steps 701 to 703:

701, extracting the associated data from the main database according to the screening rule corresponding to the target log.

In the embodiment of the present application, as can be seen from the foregoing description, the filtering rule corresponding to the log can filter out the SQL statement related to the data in the slave database corresponding to the log, and therefore, based on the filtering rule, the associated data related to the data in the slave database can also be further extracted from the master database.

And 702, determining first summary information of the associated data and second summary information of the synchronized target slave database data.

In the embodiment of the application, after the associated data is extracted from the master database, the associated data and the data in the synchronized target slave database are respectively processed according to a preset abstract algorithm to obtain the first abstract information and the second abstract information, and since the abstract information obtained after the same data is processed by the abstract algorithm is the same and the abstract information obtained after different data is processed by the abstract algorithm is different, it can be subsequently determined whether the data in the slave database is the same as the corresponding associated data in the master database by using the first abstract information and the second abstract information, that is, whether the synchronization effect is achieved is determined. And the data is processed by adopting a summary algorithm, so that the calculated amount in the checking process can be effectively reduced.

703, if the first summary information is different from the second summary information, updating the synchronized slave database according to the associated data.

In the embodiment of the application, if the first summary information is the same as the second summary information, it indicates that the data in the slave database is the same as the corresponding associated data in the master database, and at this time, it may be considered that synchronization is completed, and there is no exception. On the contrary, if the first summary information is different from the second summary information, it is indicated that the data in the slave database is different from the corresponding associated data in the master database, and an abnormality occurs in the synchronization process, the synchronized slave database needs to be updated for the second time by using the associated data in the master database.

The embodiment of the application provides a scheme for extracting the associated data from the master database by using the screening rule corresponding to the target log and verifying the synchronization process based on the summary information of the associated data and the summary information of the data in the synchronized slave database, so that the problem of abnormal data synchronization can be effectively avoided while the calculation amount is saved.

In order to better and clearly understand the complete implementation process of the data synchronization method provided in the embodiments of the present application, a complete embodiment of the data synchronization method will be given below with reference to the embodiments provided in fig. 1 to fig. 7, and the following details are given.

1) When a system is accessed to the data synchronization device from a database, the data synchronization device creates a corresponding initial log and displays a preset configuration interface of the initial log;

2) Determining a target screening rule according to rule setting operation on a preset configuration interface, and associating the target screening rule and identification information of a slave database with an initial log to obtain a candidate log;

3) After candidate logs corresponding to each slave database are formed, the data synchronization device acquires SQL statements received by the master database through an interceptor;

4) Analyzing the SQL statement to obtain the instruction type of the SQL statement; if the instruction type is a data control language, discarding the SQL statement, and if the instruction type is a data manipulation language or a data definition language, respectively processing the SQL statement according to the screening rules corresponding to the candidate logs, so as to determine a target log associated with the SQL statement in the candidate logs, and writing the SQL statement into the corresponding target log;

5) And judging whether a data synchronization request is received or not, or judging whether the number of SQL sentences written in each log is greater than a preset threshold value or not. If not, returning to the step 3) to continuously acquire the SQL statements received by the main database through the interceptor; if yes, determining a slave database corresponding to each log according to the preset database identification corresponding to each log;

6) Respectively updating each slave database according to the SQL sentences written in each log, and deleting the SQL sentences written in each log after the updating is finished;

7) Aiming at each slave database, extracting the associated data from the master database according to the screening rule of the corresponding target log to obtain the associated data;

8) Determining first summary information of the associated data and second summary information of the synchronized slave database data;

9) And judging whether the first summary information and the second summary information are the same. If the SQL statements are the same, returning to the step 3) to continuously acquire the SQL statements received by the main database through the interceptor; and if the difference is not the same, updating the synchronized slave database according to the associated data, and then returning to the step 3) to continuously acquire the SQL statements received by the master database through the interceptor.

In order to better implement the data synchronization method provided by the embodiment of the present application, on the basis of the data synchronization method, a data synchronization device is further provided in the embodiment of the present application. As shown in fig. 8, fig. 8 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application. Specifically, the data synchronization device includes:

an obtaining module 801, configured to obtain an SQL statement received by a main database;

a determining module 802, configured to write the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule;

and the synchronization module 803 is configured to update the target slave database corresponding to the target log according to the SQL statements written in the target log, so as to obtain a synchronized target slave database.

In some embodiments of the present application, the determining module includes:

the screening secondary module is used for respectively screening the SQL sentences according to the screening rules corresponding to the preset candidate logs to obtain the screening results of the preset candidate logs;

and the writing-in secondary module is used for taking the candidate log with the screening result of passing in the preset candidate logs as a target log corresponding to the SQL statement and writing the SQL statement into the target log.

In some embodiments of the present application, the data synchronization apparatus further includes a configuration module, where the configuration module includes:

the system comprises a creating secondary module, a configuration processing module and a display module, wherein the creating secondary module is used for creating an initial log and displaying a preset configuration interface of the initial log;

the selection secondary module is used for setting operation according to the rule on the preset configuration interface and determining a target screening rule;

and the setting secondary module is used for setting the target screening rule as the screening rule corresponding to the initial log and taking the set initial log as a candidate log.

In some embodiments of the present application, the synchronization module includes:

the judgment secondary module is used for determining a target slave database corresponding to the target log according to a preset database identifier corresponding to the target log if a data synchronization request is received or the number of SQL statements written in the target log is greater than a preset threshold value;

and the update secondary module is used for updating the target slave database according to the SQL statements written in the target log.

In some embodiments of the present application, the data synchronization apparatus includes:

the analysis module is used for analyzing the SQL statement to obtain the instruction type of the SQL statement;

the determining module is configured to, if the instruction type is a data manipulation language or a data definition language, write the SQL statement into a target log corresponding to the SQL statement according to a preset screening rule.

and the deleting module is used for deleting the SQL statements written in the target log.

In some embodiments of the present application, the data synchronization apparatus includes a verification module, where the verification module includes:

the extraction secondary module is used for extracting the associated data from the main database according to the screening rule corresponding to the target log;

the abstract secondary module is used for determining first abstract information of the associated data and second abstract information of the synchronized target slave database data;

and the comparison secondary module is used for updating the synchronized slave database according to the associated data if the first abstract information is different from the second abstract information.

An embodiment of the present application further provides a data synchronization device, as shown in fig. 9, fig. 9 is a schematic structural diagram of the data synchronization device provided in the embodiment of the present application.

The data synchronization device comprises a memory, a processor and a data synchronization program which is stored in the memory and can run on the processor, and the processor implements the steps of the data synchronization method provided by any embodiment of the application when executing the data synchronization program.

Specifically, the method comprises the following steps: the data synchronization apparatus may include components such as a processor 901 of one or more processing cores, a memory 902 of one or more storage media, a power supply 903, and an input unit 904. Those skilled in the art will appreciate that the data synchronization device configuration shown in fig. 9 does not constitute a limitation of the data synchronization device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 901 is a control center of the data synchronization apparatus, connects various parts of the entire data synchronization apparatus using various interfaces and lines, performs various functions of the data synchronization apparatus and processes data by running or executing software programs and/or modules stored in the memory 902 and calling the data stored in the memory 902, thereby performing overall monitoring of the data synchronization apparatus. Optionally, processor 901 may include one or more processing cores; preferably, the processor 901 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 901.

The memory 902 may be used to store software programs and modules, and the processor 901 performs various functional applications and data processing by operating the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the data synchronization apparatus, and the like. Further, the memory 902 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 access to the memory 902.

The data synchronization device further comprises a power supply 903 for supplying power to each component, and preferably, the power supply 903 may be logically connected to the processor 901 through a power management system, so that functions of managing charging, discharging, power consumption management and the like are realized through the power management system. The power supply 903 may also include any component including one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The data synchronization device may also include an input unit 904, the input unit 904 operable to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the data synchronization apparatus may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 901 in the data synchronization apparatus loads the executable file corresponding to the process of one or more application programs into the memory 902 according to the following instructions, and the processor 901 runs the application program stored in the memory 902, thereby implementing the steps in the data synchronization method provided in any embodiment of the present application.

To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like. The computer readable storage medium stores a data synchronization program, and the data synchronization program, when executed by a processor, implements the steps in the data synchronization method provided in any embodiment of the present application.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed descriptions of other embodiments, which are not described herein again.

In a specific implementation, each unit or structure may be implemented as an independent entity, or may be combined arbitrarily to be implemented as one or several entities, and the specific implementation of each unit or structure may refer to the foregoing method embodiment, which is not described herein again.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

The data synchronization method provided by the embodiment of the present application is described in detail above, and a specific example is applied in the description to explain the principle and the implementation of the present invention, and the description of the embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method of data synchronization, comprising:

acquiring an SQL statement received by a main database;

2. The data synchronization method according to claim 1, wherein before the SQL statements are respectively filtered according to the filtering rules corresponding to the preset candidate logs and the filtering results of the preset candidate logs are obtained, the method comprises:

and setting the target screening rule as a screening rule corresponding to the initial log, and taking the set initial log as a candidate log.

3. The data synchronization method according to claim 1, wherein the updating the target slave database corresponding to the target log according to the SQL statement written in the target log comprises:

4. The data synchronization method according to claim 1, before writing the SQL statement into a target log corresponding to the SQL statement according to a preset filtering rule, the method further comprises:

5. The data synchronization method according to claim 1, wherein after the target corresponding to the target log is updated from the database according to the SQL statements written in the target log, the method further comprises:

and deleting the SQL statements written in the target log.

6. The data synchronization method according to claim 1, wherein after the target slave database corresponding to the target log is updated according to the SQL statement written in the target log and the synchronized target slave database is obtained, the method further comprises:

7. The data synchronization method according to any one of claims 1 to 6, wherein the master database and the target slave database are relational databases.

8. A data synchronization apparatus, comprising:

9. A data synchronization device, characterized in that the data synchronization device comprises a processor, a memory and a data synchronization program stored in the memory and executable on the processor, the processor executing the data synchronization program to implement the steps in the data synchronization method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a data synchronization program, the data synchronization program being executed by a processor to implement the steps in the data synchronization method of any one of claims 1 to 7.