WO2022001750A1

WO2022001750A1 - Data synchronization processing method, electronic device, and storage medium

Info

Publication number: WO2022001750A1
Application number: PCT/CN2021/101542
Authority: WO
Inventors: 李磊; 朱业
Original assignee: 中兴通讯股份有限公司
Priority date: 2020-06-29
Filing date: 2021-06-22
Publication date: 2022-01-06
Also published as: CN113934745A

Abstract

Provided are a data synchronization processing method, an electronic device, and a storage medium. The method comprises: receiving, by a master node, an SQL statement sent by a client (S100); generating an SQL execution plan (S110a); sending the SQL execution plan to slave nodes (S110b); executing the SQL execution plan (S120); obtaining a first incremental log of the master node, and performing Hash calculation on data of the first incremental log to obtain a first Hash value (S130); receiving a second Hash value sent by each slave node (S140); and determining a synchronization result according to the first Hash value and the second Hash value (S150).

Description

Data synchronization processing method, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202010609672.8 and the filing date of June 29, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The present application relates to the technical field of distributed databases, and in particular, to a data synchronization processing method, an electronic device and a storage medium.

Background technique

The existing data synchronization schemes of distributed databases mainly include asynchronous replication, full synchronous replication and semi-synchronous replication. Among them, the asynchronous replication scheme has the least impact on the execution efficiency of the database, but it is prone to data inconsistency between the master node and the slave node; although the full synchronous replication scheme can ensure the consistency of data synchronization between the master node and the slave node, it has a The performance of the distributed database system has a great impact and the efficiency is low, and it is not suitable for high-concurrency application scenarios. Although the semi-synchronous replication scheme takes into account the efficiency and data consistency, there is still at least one incremental log synchronization disk placement cycle. Data delay .

SUMMARY OF THE INVENTION

The following is an overview of the topics detailed in this article. This summary is not intended to limit the scope of protection of the claims.

Embodiments of the present application provide a data synchronization processing method, an electronic device, and a storage medium.

On the one hand, an embodiment of the present application provides a data synchronization processing method, which is applied to a master node. The method includes: receiving a structured query language SQL statement sent by a client; parsing the SQL statement and generating an SQL execution plan , and send the SQL execution plan to at least one slave node; execute the SQL execution plan; obtain the first incremental log generated when the master node executes the SQL execution plan; Perform hash calculation on the data to obtain a first hash value; receive a second hash value sent by each of the slave nodes, wherein the second hash value is the execution plan for the slave node to execute the SQL The data of the second incremental log generated during the hash calculation is obtained; the synchronization result is determined according to the first hash value and the second hash value; the operation result information is sent to the client according to the synchronization result .

On the other hand, an embodiment of the present application provides a data synchronization processing method, which is applied to a master node. The method includes: receiving a structured query language SQL statement sent by a client; sending the SQL statement to at least one slave node ;

Parse the SQL statement and generate a SQL execution plan; execute the SQL execution plan; obtain the first incremental log generated by the master node executing the SQL execution plan, and perform the data analysis on the first incremental log. Hash calculation to obtain a first hash value; receiving a second hash value sent by each of the slave nodes, wherein the second hash value is to execute the SQL generated according to the SQL statement on the slave node The data of the second incremental log generated by the execution plan is obtained by hash calculation; the synchronization result is determined according to the first hash value and the second hash value; the operation result information is sent to the client according to the synchronization result end.

On the other hand, an embodiment of the present application provides a data synchronization processing method, which is applied to a slave node. The method includes: receiving an SQL execution plan sent by a master node; executing the SQL execution plan; The second incremental log generated by the SQL execution plan, perform hash calculation on the data of the second incremental log to obtain a second hash value; send the second hash value to the master node, so that The master node determines the synchronization result according to the first hash value and the second hash value, wherein the first hash value performs data on the incremental log generated by the master node executing the SQL execution plan. Hash is calculated.

On the other hand, an embodiment of the present application provides a data synchronization processing method, which is applied to a slave node. The method includes: receiving an SQL statement sent by a master node; parsing the SQL statement and generating an SQL execution plan; the SQL execution plan; obtain the second incremental log generated by the execution of the SQL execution plan by the slave node, perform hash calculation on the data of the second incremental log, and obtain a second hash value; send the second incremental log Two hash values are sent to the master node, so that the master node determines the synchronization result according to the first hash value and the second hash value, wherein the first hash value performs all the operations on the master node. The data of the incremental log generated by the SQL execution plan is obtained by hash calculation.

On the other hand, an embodiment of the present application provides an electronic device, including: a memory for storing a program; a processor for executing the program stored in the memory, when the processor executes the program stored in the memory , the processor is configured to execute any one of the data synchronization processing methods described above.

In another aspect, an embodiment of the present application provides a storage medium storing computer-executable instructions, where the computer-executable instructions are used to execute any of the data synchronization processing methods described above.

Other features and advantages of the present application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the description, claims and drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification. They are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

1 is a flowchart of a data synchronization processing method applied to a master node provided by an embodiment of the present application;

2a is a flowchart of another data synchronization processing method applied to a master node provided by an embodiment of the present application;

2b is a flowchart of another data synchronization processing method applied to a master node provided by an embodiment of the present application;

3 is a flowchart of another data synchronization processing method applied to a master node provided by an embodiment of the present application;

4 is a flowchart of another data synchronization processing method applied to a master node provided by an embodiment of the present application;

5 is a flowchart of a data synchronization processing method applied to a slave node provided by an embodiment of the present application;

6 is a flowchart of another data synchronization processing method applied to a slave node provided by an embodiment of the present application;

7 is a flowchart of another data synchronization processing method applied to a slave node provided by an embodiment of the present application;

8 is a flowchart of another data synchronization processing method applied to a slave node provided by an embodiment of the present application;

9 is a schematic diagram of an application scenario of a data synchronization processing method provided by an embodiment of the present application;

FIG. 10 is an exemplary flowchart of a data synchronization processing method provided by an embodiment of the present application applied to the scenario shown in FIG. 9;

FIG. 11 is an exemplary flowchart of another data synchronization processing method provided by an embodiment of the present application applied to the scenario shown in FIG. 9;

FIG. 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

detailed description

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

It should be understood that in the description of the embodiments of the present application, multiple (or multiple) means more than two, greater than, less than, exceeding, etc. are understood as not including this number, above, below, within, etc. are understood as including this number. If there is a description of "first", "second", etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance, or implicitly indicating the number of indicated technical features or implicitly indicating the indicated The sequence of technical characteristics.

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the embodiments of this application, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that, in the description of this application, unless otherwise specified, "/" indicates that the associated objects are in an "or" relationship, for example, A/B can indicate A or B; in this application, "and" "/or" is just an association relationship that describes an associated object, which means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. where A and B can be singular or plural.

The existing data synchronization schemes of distributed databases mainly include asynchronous replication, full synchronous replication and semi-synchronous replication.

The asynchronous replication scheme is that the master node returns the execution result to the client immediately after executing the transaction submitted by the client, and does not care about the data gap between the slave node and the master node and the degree of data playback, so that the master node and the slave node are prone to appear. Data inconsistency between nodes. If the master node crashes, the transactions submitted by the master node may not be transmitted to the slave node. If the slave node is forcibly promoted to the master node at this time, the data on the new master node may be incomplete.

The full synchronous replication scheme means that after the master node completes the execution of the transaction submitted by the client, it sends the incremental log generated by executing the transaction to each slave node, and the slave node parses and replays the received incremental operation log and then submits the transaction. Wait until all slave nodes have executed the transaction commit before returning the execution result to the client. Because it is necessary to wait for all slave nodes to complete the execution of the transaction before returning the execution result to the client, the system performance of full synchronous replication will inevitably be seriously affected.

The semi-synchronous replication scheme is between asynchronous replication and full synchronous replication. After the master node completes the execution of the transaction submitted by the client, it sends the incremental log generated by the execution of the transaction to each node, and waits for at least one node to receive the incremental log and copy it. After the incremental log is successfully written to the relay log, the result is returned to the client. Although the semi-synchronous replication scheme takes into account both efficiency and data consistency, there is still a data delay of at least one incremental log synchronization disk placement cycle.

To this end, the embodiments of the present application provide a data synchronization processing method, an electronic device, and a storage medium, which can ensure the consistency of data synchronization and improve the efficiency of data synchronization.

FIG. 1 shows a flowchart of a data synchronization processing method provided by an embodiment of the present application. As shown in FIG. 1 , the data synchronization processing method is applied to the master node of the distributed database system, and the method includes but is not limited to the following steps S100 to S150 at least.

Step S100, the master node receives a structured query language (Structured Query Language, SQL for short) statement sent by the client.

It should be understood that the SQL statement sent by the client in the embodiment of the present application is a statement used to describe the transaction submitted by the client.

Step S110a, the master node parses the SQL statement and generates an SQL execution plan.

Step S110b, sending the SQL execution plan to at least one slave node.

In this embodiment of the present application, after receiving the SQL statement sent by the client, the master node parses the SQL statement and generates an SQL execution plan. Then, the SQL execution plan sent by the client is synchronized to one or more slave nodes, so that each slave node executes the SQL execution plan.

Step S120, the master node executes the SQL execution plan.

It should be understood that, in specific implementation, while the master node executes the SQL execution plan, each slave node is also executing the SQL execution plan forwarded by the master node, so as to achieve the purpose of executing the SQL execution plan in parallel by the master node and the slave nodes.

Step S130, the master node acquires the first incremental log generated by the master node executing the SQL execution plan, and performs hash calculation on the data of the first incremental log to obtain a first hash value.

Exemplarily, when the master node executes the SQL execution plan, corresponding transaction log information is generated to record relevant operation information when the master node executes the SQL execution plan. The above transaction log information is usually written into the operation log of the master node, which is usually stored in the database storage engine of the master node. After the transaction log information is written into the operation log of the master node, the newly added log content in the operation log of the master node is the above-mentioned first incremental log. In this embodiment of the present application, a hash algorithm is used to calculate and process the data of the first incremental log to obtain a first hash value.

Step S140, the master node receives the second hash value sent by each slave node, where the second hash value is obtained by hashing the data of the second incremental log generated by the slave node executing the SQL execution plan.

Exemplarily, after receiving the SQL execution plan synchronously sent by the master node, the slave node executes the SQL execution plan. When the slave node executes the SQL execution plan, the corresponding transaction log information is also generated and written into the operation log of the slave node. The slave node obtains the second incremental log from its own operation log, and uses a hash algorithm to calculate and process the data of the second incremental log to obtain a second hash value. In this embodiment of the present application, each slave node sends the second hash value calculated by the slave node to the master node, so that the master node performs subsequent processing on the first hash value and the second hash value.

Step S150, the master node determines the synchronization result according to the first hash value and the second hash value.

Exemplarily, the master node performs a hash comparison between the first hash value and the second hash value received from each slave node, and if the comparison result is that the first hash value is equal to each second hash value, it indicates that The data after the master node and each slave node execute the SQL execution plan are consistent, that is, the data of the master node and each slave node are successfully synchronized; if the comparison result is that there are one or more second hash values that are inconsistent with the first hash value If they are equal, it indicates that the data after one or more slave nodes execute the SQL execution plan is inconsistent with the data after the master node executes the SQL execution plan, that is, the data synchronization of the one or more slave nodes fails.

Step S160, sending operation result information to the client according to the synchronization result.

During specific implementation, the operation result may be determined according to the number of slave nodes whose data synchronization is successful. For example, in one example, as long as the number of successfully synchronized slave nodes meets a preset threshold, it is considered that the transaction submitted by the client is executed successfully, and operation success information is returned to the client. For example, in another example, when the data synchronization of any node fails, it is considered that execution of the transaction submitted by the client fails, and operation failure information is returned to the client. It should be understood that those skilled in the art can customize specific conditions for determining a successful operation according to a synchronization result according to requirements, which are not specifically limited in this embodiment of the present application.

As shown in FIG. 2a, in some embodiments, the following steps S111a and S112a are further included after step S110b.

Step S111a, the master node starts a timer, and when the timer does not expire, receives the first reply information sent by the slave node, wherein the first reply information indicates that the slave node successfully or failed to receive the SQL execution plan;

Step S112a, the master node determines whether the number of slave nodes that successfully receive the SQL execution plan meets the first preset threshold according to the received first reply information.

Exemplarily, after the master node sends the SQL execution plan to each slave node, the timer is started to start the timing. When the timer expires and there are slave nodes that have not sent the first reply information, it can be determined that the slave node that has not sent the first reply information The slave node fails to receive the SQL execution plan, so it can be determined that the slave node that has not sent the first reply information fails to perform data synchronization. In the embodiment of the present application, the timer is used for timing, so as to avoid the increase of data delay caused by waiting for the reply information for too long.

In this embodiment of the present application, the master node performs the subsequent step S120 only when the number of slave nodes that successfully receive the SQL execution plan meets the first preset threshold. Specifically, the number of slave nodes that successfully receive the SQL execution plan can be determined according to the first reply information sent by each slave node and the number of slave nodes that have not sent the first reply information when the timer expires.

As an example, the master node sends the SQL execution plan to the 4 slave nodes respectively, and then starts the timer. When the timer expires, only 3 first reply messages are received from the slave nodes, and all the received first reply messages indicate that the corresponding slave node successfully received the SQL execution plan, indicating that one node failed to receive the SQL execution plan. . Assume that the first preset threshold is 2 and the number of slave nodes that currently successfully receive the SQL execution plan is required to be greater than or equal to the first preset threshold, that is, the number of slave nodes that currently successfully receive the SQL execution plan meets the first preset threshold, and the master node continues Step S120 is performed. Of course, the first preset threshold may also be set to be equal to the number of slave nodes, that is, all current slave nodes are required to successfully receive the SQL execution plan before the subsequent step S120 is allowed to be executed.

As shown in FIG. 2b, in some embodiments, after step S120, it further includes:

Step S121, the master node receives the second reply information sent by the slave node, and determines the number of slave nodes that successfully execute the SQL execution plan according to the received second reply information, wherein the second reply information indicates that the slave node successfully executed the SQL execution plan or Fail;

Step S122, the master node determines whether the master node successfully executes the SQL execution plan and the number of nodes that successfully execute the SQL execution plan meets the second preset threshold.

During specific implementation, if one or more nodes fail to execute the SQL execution plan, the one or more nodes that fail to execute the SQL execution plan send second reply information to the master node. The master node determines the number of nodes that successfully execute the SQL execution plan according to the number of received second reply information.

In this embodiment of the present application, only when the master node successfully executes the SQL execution plan and the number of nodes that successfully execute the SQL execution plan meets the second preset threshold, the subsequent step S130 is allowed to be executed. If the master node fails to execute the SQL execution plan or the number of slave nodes that successfully execute the SQL execution plan does not meet the second preset threshold, the execution of the subsequent steps is terminated, and the operation failure information is directly sent to the client.

As an example, the master node sends SQL execution plans to 4 nodes respectively, and the 4 slave nodes all receive the SQL execution plans successfully. When the master node and each slave node execute the SQL execution plan in parallel, there is one slave node that fails to execute the SQL execution plan and feeds back the second reply information indicating the failure to execute the SQL execution plan to the master node. Assuming that the second preset threshold is 2 and the number of slave nodes that currently successfully execute the SQL execution plan is required to be greater than or equal to the second preset threshold, that is, the number of slave nodes that currently successfully execute the SQL execution plan meets the second preset threshold, so the master node allows The subsequent step S130 is performed. Of course, the second preset threshold may also be set to be equal to the number of slave nodes, that is, the subsequent step S130 is allowed to be executed only after all current slave nodes are required to successfully execute the SQL execution plan.

As shown in FIG. 3, in another embodiment, step S110a can be replaced with step S110c, the master node sends the SQL statement to at least one slave node; step S110b can be replaced with step S110d, the master node parses the SQL statement and Generate SQL execution plan.

In this embodiment, the master node directly sends the received SQL statement from the client to the slave node. After receiving the SQL statement sent by the master node, the slave node parses the SQL statement and generates an SQL execution plan; then the slave node executes the SQL execution plan.

It should be understood that the order of step S110c and step S110d may be interchanged.

As shown in FIG. 4 , after step S110c, it may further include: step S111b, the master node starts a timer, and if the timer does not time out, receives the first reply information sent by the slave node, wherein the first reply information indicates the slave node. The node receives the SQL statement successfully or failed.

Step S111b includes step S112b, in which the master node determines, according to the received first reply information, whether the number of slave nodes that successfully receive the SQL statement meets a first preset threshold.

Exemplarily, after the master node sends the SQL statement to each slave node, the timer is started to start the timing, and when the timer expires and there are slave nodes that have not sent the first reply information, it can be determined that the slave nodes that have not sent the first reply information If the node fails to receive the SQL statement, it may be determined that the slave node that has not sent the first reply information fails to perform data synchronization. In the embodiment of the present application, the timer is used for timing, so as to avoid the increase of data delay caused by waiting for the reply information for too long.

In this embodiment of the present application, the master node performs the subsequent steps only when the number of slave nodes that successfully receive the SQL statement meets the first preset threshold. Specifically, the number of slave nodes that successfully receive the SQL statement can be determined according to the first reply information sent by each slave node and the number of slave nodes that have not sent the first reply information when the timer expires.

FIG. 5 shows a flowchart of a data synchronization processing method provided by an embodiment of the present application. As shown in Fig. 5, the data synchronization processing method is applied to a node of a distributed database system, and the method at least includes but is not limited to the following steps S200a to S230.

Step S200a, the slave node receives the SQL execution plan sent by the master node.

Exemplarily, after receiving the SQL execution plan sent by the client, the master node synchronizes the SQL execution plan sent by the client to the slave node.

Step S210, the slave node executes the SQL execution plan.

Exemplarily, after receiving the SQL execution plan from the master node, the slave node can parse and execute the SQL execution plan, implement the SQL execution plan in parallel with the master node, and replicate the data of the master node to the slave node synchronously.

Step S220: Obtain the second incremental log generated by the node executing the SQL execution plan from the node, perform hash calculation on the data of the second incremental log, and obtain a second hash value.

Exemplarily, when the slave node executes the SQL execution plan, corresponding transaction log information is generated to record relevant operation information when the slave node executes the SQL execution plan. The above transaction log information is usually written into the operation log of the slave node, and usually, the operation log is stored in the database storage engine of the slave node. After the transaction log information is written into the operation log of the slave node, the newly added log content in the operation log of the slave node is the above-mentioned second incremental log. In this embodiment of the present application, a hash algorithm is used to calculate and process the data of the second incremental log of the slave node to obtain a second hash value.

Step S230, the slave node sends the second hash value to the master node, so that the master node determines the synchronization result according to the first hash value and the second hash value.

The first hash value is obtained by performing hash calculation on the data of the first incremental log generated by the master node executing the SQL execution plan by itself in step S130.

Exemplarily, in this embodiment of the present application, the slave node sends the second hash value to the master node, and after receiving the second hash value from the slave node, the master node compares the first hash value with the second hash value. Perform a hash comparison. If the comparison result is that the first hash value is equal to the second hash value, it indicates that the data after the master node and the slave node execute the SQL execution plan are consistent, that is, the data of the master node and the slave node are synchronized. Success; if the comparison result is that the first hash value is not equal to the second hash value, it indicates that the data after the SQL execution plan is executed by the slave node is inconsistent with the data after the master node executes the SQL execution plan, that is to say, the slave node is synchronized The data for the primary node failed.

As shown in FIG. 6 , in some instances, after step S200a, it further includes: step S201a, sending first reply information to the master node, wherein the first reply information indicates that the slave node succeeds or fails to receive the SQL execution plan.

After the slave node finishes receiving the SQL execution plan sent by the master node, it sends the first reply message to the master node, so that the master node determines whether the slave node accepts the SQL execution plan successfully, and then judges whether to execute the subsequent steps.

As shown in FIG. 6 , in some instances, after step S210 , the method further includes: step S211 , sending second reply information to the master node, wherein the second reply information indicates that the slave node succeeds or fails to execute the SQL execution plan.

After the slave node finishes executing the SQL execution plan, it sends the second reply information to the master node, so that the master node determines whether the slave node successfully executes the SQL execution plan, and then determines whether to execute the subsequent steps.

As shown in FIG. 7, step S200a can be replaced with the following steps S200b and S200c:

S200b, the slave node receives the SQL statement sent by the master node;

S200c, parse the SQL statement and generate an SQL execution plan.

In some possible embodiments, the master node directly forwards the SQL statement sent by the client to the slave node. After the slave node receives the SQL statement sent by the master node, the SQL statement needs to be parsed to generate an SQL execution plan, and then the subsequent step S210 is performed.

As shown in FIG. 8 , step S201b may be included between steps S200b and S200c, sending first reply information to the master node, wherein the first reply information indicates that the slave node succeeds or fails in receiving the SQL statement.

The data synchronization processing method provided by the embodiments of the present application will be further described below with reference to specific examples. As shown in FIG. 9 , in an application scenario, the distributed database system includes a client 10 , a master node 20 and a slave node 30 . When the data synchronization processing method provided by the embodiment of the present application is applied to the scenario shown in FIG. 10 , it may specifically include the exemplary steps S300 to S309 shown in FIG. 10 .

Step S300, the master node 20 receives the SQL statement sent by the client 10.

Step S301, the master node 20 parses the SQL statement and generates an SQL execution plan. It should be understood that the master node 20 executes the subsequent step S302 only when the parsing of the SQL statement is successful; otherwise, the operation failure information is sent to the client 10 .

Step S302, the master node 20 sends the SQ execution plan to the slave node 30.

Step S303 , after the slave node 30 successfully receives the SQL execution plan sent by the master node 20 , it sends first reply information to the master node 20 to indicate that it has successfully received the SQL execution plan.

Step S304a, the master node 20 executes the SQL execution plan;

Step S304b, execute the SQL execution plan from the node 30;

Step S305 , the slave node 30 sends the second reply information to the master node 20 to indicate that it has successfully executed the SQL execution plan.

Step S306a, the master node 20 obtains the first incremental log generated by its own execution of the SQL execution plan, performs hash calculation on the data of the first incremental log, and obtains the first hash value;

Step S306b, obtaining the second incremental log generated by executing the SQL execution plan by itself from the node 30, and performing hash calculation on the data of the second incremental log to obtain a second hash value;

Step S307 , the slave node 30 sends the second hash value to the master node 20 .

Step S308, the master node 20 compares the first hash value with the second hash value, and determines a synchronization result according to the comparison result.

Step S309, the master node 20 sends the operation result information to the client 10 according to the synchronization result. Specifically, if the synchronization result is that the synchronization is successful, the operation success information is sent to the client 10 , otherwise, the operation failure information is sent to the client 10 .

It should be understood that the above-mentioned steps S304a and S304b are steps that can be executed in parallel; steps S306a and S306b are steps that can be executed in parallel.

When the data synchronization processing method provided by the embodiment of the present application is applied to the scenario shown in FIG. 9 , it may further include the exemplary steps S400 to S410 shown in FIG. 11 .

Step S400, the master node 20 receives the SQL statement sent by the client 10.

Step S401 , the master node 20 forwards the SQL statement to the slave node 30 .

Step S402 , after the slave node 30 successfully receives the SQL statement sent by the master node 20 , it sends the first reply message to the master node 20 to indicate that it has successfully received the SQL statement.

Step S403a, the master node 20 parses the SQL statement and generates an SQL execution plan.

Step S403b, the slave node 30 parses the SQL statement and generates an SQL execution plan.

Step S404a, the master node 20 executes the SQL execution plan;

Step S404b, execute the SQL execution plan from the node 30;

Step S405 , the slave node 30 sends the second reply message to the master node 20 to indicate that it has successfully executed the SQL execution plan.

Step S406a, the master node 20 obtains the first incremental log generated by its own execution of the SQL execution plan, performs hash calculation on the data of the first incremental log, and obtains the first hash value;

Step S406b, obtaining the second incremental log generated by executing the SQL execution plan by itself from the node 30, and performing hash calculation on the data of the second incremental log to obtain a second hash value;

Step S407 , the slave node 30 sends the second hash value to the master node 20 .

Step S408, the master node 20 compares the first hash value with the second hash value, and determines a synchronization result according to the comparison result.

Step S409, the master node 20 sends the operation result information to the client 10 according to the synchronization result. Specifically, if the synchronization result is that the synchronization is successful, the operation success information is sent to the client 10 , otherwise, the operation failure information is sent to the client 10 .

In the traditional data synchronization scheme, after the master node completes the execution of the SQL execution plan, the corresponding incremental log is sent to the node for analysis and playback, which is equivalent to a serial data replication method. According to the solution of the embodiment of the present application, the master node and the node execute the SQL execution plan synchronously, which is equivalent to parallel data replication between the master node and the node. Compared with the traditional data synchronization solution, the solution of the embodiment of the present application can greatly improve the database performance. The efficiency of synchronous processing reduces the delay of data, improves the performance of the distributed database system, reduces the processing time for the client to submit transactions, and further improves the user experience.

In addition, the embodiment of the present application compares the incremental logs of the master node and the node through the Habi algorithm, and determines the synchronization result according to the comparison result, which can ensure the consistency of data synchronization.

FIG. 12 shows an electronic device 40 provided by an embodiment of the present application. As shown in Figure 12, the electronic device 40 includes but is not limited to:

memory 42 for storing programs;

The processor 41 is configured to execute the program stored in the memory 42. When the processor 41 executes the program stored in the memory 42, the processor 41 is configured to execute the data synchronization processing method of any of the foregoing embodiments.

The processor 41 and the memory 42 may be connected by a bus or other means.

As a non-transitory computer-readable storage medium, the memory 42 can be used to store non-transitory software programs and non-transitory computer-executable programs, such as any data synchronization processing method described in the embodiments of this application. The processor 41 implements the data synchronization processing method in any of the foregoing embodiments by running the non-transitory software programs and instructions stored in the memory 42 .

The memory 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required by at least one function; the storage data area may store and execute the above-mentioned data synchronization processing method. Additionally, memory 42 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 42 may include memory located remotely from processor 41, which may be connected to processor 41 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The non-transitory software programs and instructions required to implement the above-mentioned data synchronization processing method are stored in the memory 42 , and when executed by one or more processors 41 , execute the data synchronization processing method of any of the above-mentioned embodiments.

Embodiments of the present application further provide a storage medium storing computer-executable instructions, where the computer-executable instructions are used to execute the data synchronization processing method of any of the foregoing embodiments.

In one embodiment, the storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors 41. One or more processors 41 execute the data synchronization processing method of any of the above embodiments.

The embodiments of the present application include: the master node receives the structured query language SQL statement sent by the client; the master node parses the SQL statement and generates an SQL execution plan, and then sends the SQL execution plan to at least one slave node, or directly Send the SQL statement to at least one slave node; the master node executes the SQL execution plan; the master node obtains the first incremental log generated by the master node executing the SQL execution plan, and the first incremental log is Hash the data to obtain the first hash value; the master node receives the second hash value sent by each of the slave nodes, wherein the second hash value is the execution of the SQL on the slave node. The data of the second incremental log generated by the execution plan is obtained by hash calculation; the master node determines the synchronization result according to the first hash value and the second hash value. In the solution of the embodiment of the present application, on the one hand, the master node and the slave node execute the SQL execution plan at the same time, which realizes the parallel synchronous replication of the master node and the slave node, greatly improves the efficiency of database synchronization processing, and reduces the delay of data; On the other hand, the incremental log of the master node and the node is compared through the Habi algorithm, and the synchronization result is determined according to the comparison result, which can ensure the consistency of data synchronization.

The above-described embodiments are only illustrative, and the units described as separate components may or may not be physically separated, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

Those of ordinary skill in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data flexible, removable and non-removable media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or may Any other medium used to store desired information and which can be accessed by a computer. In addition, communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery media, as is well known to those of ordinary skill in the art .

The above is a specific description of some implementations of the application, but the application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent deformations or replacements under the shared conditions that do not violate the scope of the application. Equivalent modifications or substitutions are included within the scope defined by the claims of the present application.

Claims

A data synchronization processing method, applied to a master node, the method comprising:

Receive the structured query language SQL statement sent by the client;

Parse the SQL statement and generate an SQL execution plan;

sending the SQL execution plan to at least one slave node;

Execute the SQL execution plan;

Obtain the first incremental log generated when the master node executes the SQL execution plan, perform hash calculation on the data of the first incremental log, and obtain a first hash value;

Receive a second hash value sent by each of the slave nodes, where the second hash value is a hash calculation performed on the data of the second incremental log generated when the slave node executes the SQL execution plan get;

determining a synchronization result according to the first hash value and the second hash value;

Send operation result information to the client according to the synchronization result.
The data synchronization processing method according to claim 1, wherein when the SQL execution plan is sent to at least one slave node, the method further executes:

start the timer;

If the timer does not expire, the first reply information sent by the slave node is received, wherein the first reply information indicates that the slave node succeeds or fails in receiving the SQL execution plan.
The data synchronization processing method according to claim 2, wherein the executing the SQL execution plan comprises:

In the case that the number of the slave nodes that successfully receive the SQL execution plan meets a first preset threshold, execute the SQL execution plan.
The data synchronization processing method according to claim 1, wherein when executing the SQL execution plan, the second reply information sent by the slave node is also received, and it is determined according to the received second reply information that the execution of the SQL execution plan is successful. The number of slave nodes, wherein the second reply information indicates that the slave node succeeds or fails to execute the SQL execution plan.
The data synchronization processing method according to claim 4, wherein the acquiring the first incremental log generated by the master node executing the SQL execution plan comprises:

In the case that the master node successfully executes the SQL execution plan and the number of the nodes that successfully execute the SQL execution plan meets a second preset threshold, acquire the first result generated by the master node executing the SQL execution plan Incremental log.
A data synchronization processing method, applied to a master node, the method comprising:

Receive the structured query language SQL statement sent by the client;

sending the SQL statement to at least one slave node;

Parse the SQL statement and generate an SQL execution plan;

Execute the SQL execution plan;

obtaining the first incremental log generated by the master node executing the SQL execution plan, and performing hash calculation on the data of the first incremental log to obtain a first hash value;

Receive a second hash value sent by each of the slave nodes, where the second hash value is data of the second incremental log generated by the slave node executing the SQL execution plan generated according to the SQL statement Hash calculation is performed;

determining a synchronization result according to the first hash value and the second hash value;

Send operation result information to the client according to the synchronization result.
The data synchronization processing method according to claim 6, wherein, when the SQL statement is sent to at least one slave node, it also executes:

start the timer;

If the timer does not expire, the first reply information sent by the slave node is received, wherein the first reply information indicates that the slave node succeeds or fails in receiving the SQL statement.
The data synchronization processing method according to claim 7, wherein the executing the SQL execution plan comprises:

In the case that the number of the slave nodes that successfully receive the SQL statement meets a first preset threshold, the SQL execution plan is executed.
The data synchronization processing method according to claim 6, wherein when executing the SQL execution plan, the second reply information sent by the slave node is also received, and it is determined to execute the SQL execution according to the received second reply information The number of slave nodes whose plan is successful, wherein the second reply information indicates that the slave node succeeds or fails to execute the SQL execution plan.
The data synchronization processing method according to claim 9, wherein the acquiring the first incremental log generated by the master node executing the SQL execution plan comprises:

In the case that the master node successfully executes the SQL execution plan and the number of the nodes that successfully execute the SQL execution plan meets a second preset threshold, acquire the first result generated by the master node executing the SQL execution plan Incremental log.
A data synchronization processing method, applied to a slave node, the method comprising:

Receive the SQL execution plan sent by the master node;

Execute the SQL execution plan;

Obtaining the second incremental log generated by the execution of the SQL execution plan by the slave node, and performing hash calculation on the data of the second incremental log to obtain a second hash value;

Sending the second hash value to the master node, so that the master node determines a synchronization result according to the first hash value and the second hash value, wherein the first hash value is associated with the The data of the incremental log generated by the master node executing the SQL execution plan is obtained by hash calculation.
The data synchronization processing method according to claim 11, wherein when receiving the SQL statement sent by the master node, first reply information is also sent to the master node, wherein the first reply information instructs the slave node to receive all the data. Specifies whether the SQL execution plan succeeds or fails.
The data synchronization processing method according to claim 11, wherein when executing the SQL statement, second reply information is also sent to the master node, wherein the second reply information instructs the slave node to execute the SQL The execution plan succeeds or fails.
A data synchronization processing method, applied to a slave node, the method comprising:

Receive the SQL statement sent by the master node;

Parse the SQL statement and generate an SQL execution plan;

Execute the SQL execution plan;

Obtaining the second incremental log generated by the execution of the SQL execution plan by the slave node, and performing hash calculation on the data of the second incremental log to obtain a second hash value;

Sending the second hash value to the master node, so that the master node determines a synchronization result according to the first hash value and the second hash value, wherein the first hash value is associated with the The data of the incremental log generated by the master node executing the SQL execution plan is obtained by hash calculation.
The data synchronization processing method according to claim 14, wherein when receiving the SQL statement sent by the master node, first reply information is also sent to the master node, wherein the first reply information instructs the slave node to receive all the data. success or failure of the SQL statement.
The data synchronization processing method according to claim 14, wherein when executing the SQL execution plan, second reply information is also sent to the master node, wherein the second reply information instructs the slave node to execute the The SQL execution plan succeeds or fails.
An electronic device comprising:

memory for storing programs;

a processor, configured to execute the program stored in the memory, wherein when the processor executes the program stored in the memory, the processor is configured to execute:

A method as claimed in any one of claims 1 to 16.
A storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute:

A method as claimed in any one of claims 1 to 16.