WO2023109238A1

WO2023109238A1 - Data synchronization method and system, and storage medium

Info

Publication number: WO2023109238A1
Application number: PCT/CN2022/120536
Authority: WO
Inventors: 陈洁娜
Original assignee: 深圳前海微众银行股份有限公司
Priority date: 2021-12-17
Filing date: 2022-09-22
Publication date: 2023-06-22
Also published as: CN114385754A

Abstract

Disclosed in the present application is a data synchronization method. The method comprises: if a target operation statement of a target operation type addressing a target data table is detected, determining at least one second storage node which has a correlation with the target data table, wherein the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; and on the basis of each data synchronization mode, sending the target operation statement to the corresponding second storage node, so as to realize a synchronous update on the target data table in the first storage node and the second storage node. Further disclosed in the present application are a data synchronization system and a storage medium.

Description

A data synchronization method, system and storage medium

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202111552168.X and a filing date of December 17, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The present application relates to the field of computer technology, in particular to a data synchronization method, system and storage medium.

Background technique

With the rapid development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, more and more technical requirements high demands. With the increasing demands of users, distributed storage systems have been widely used in order to ensure user experience. In the application scenario of a distributed storage system, when the data in a certain storage system needs to be modified, such as adding, deleting, modifying, etc., it is necessary to make corresponding changes to the corresponding copy data in other systems to ensure data consistency. .

At present, when the data in a storage system needs to be modified, and the copy data in other systems needs to be modified accordingly, the common method is to send modification requests to other systems at the same time. However, in this implementation process, each other system needs to maintain a copy of the same code. However, due to the fast iteration speed of the business system, the maintainers of each other system are usually not uniform, which leads to the inconsistency of the code jointly maintained by the system. As a result, it is impossible to respond to the modification request in time, resulting in data inconsistency in the distributed storage system, resulting in low data synchronization efficiency.

Contents of the invention

In order to solve the above technical problems, the embodiment of the present application expects to provide a data synchronization method, system and storage medium, which solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves It improves the efficiency of data synchronization in distributed storage systems.

The technical scheme of the present application is realized like this:

In a first aspect, a data synchronization method is applied to a first storage node, and the method includes:

If a target operation statement belonging to the target operation type for the target data table is detected, at least one second storage node having an association relationship with the target data table is determined; wherein the target operation statement is used to change the target data table content in

determining a data synchronization mode for each of the second storage nodes;

Based on each of the data synchronization modes, the target operation statement is sent to the corresponding second storage node, so as to realize synchronous update of the target data table in the first storage node and the second storage node.

In a second aspect, a data synchronization system, the system includes: a first storage node and at least one second storage node; wherein:

The first storage node is configured to determine at least one second storage node associated with the target data table if a target operation statement belonging to the target operation type is detected for the target data table; wherein the target operation The statement is used to change the content in the target data table; determine the data synchronization mode of each of the second storage nodes; based on each of the data synchronization modes, send the target operation statement to the corresponding second storage node node;

The second storage node is configured to receive the target operation statement; use the target operation statement to perform an update operation on the target data table corresponding to the target operation statement in the second storage node, to obtain an updated result.

In a third aspect, a storage medium stores a data synchronization program, and when the data synchronization program is executed by a processor, the steps of the data synchronization method described in any one of the foregoing are implemented.

In the embodiment of the present application, if a target operation statement belonging to the target operation type is detected for the target data table, the first storage node determines at least one second storage node that has an association relationship with the target data table, and determines that each second storage node The data synchronization method, and based on each data synchronization method, send the target operation statement to the corresponding second storage node, so that the second storage node executes the target operation statement, and realizes the synchronous update operation of the target data table with the first storage node , which solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system.

Description of drawings

FIG. 1 is a schematic flow diagram of a data synchronization method provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart of another data synchronization method provided by the embodiment of the present application;

FIG. 3 is a schematic flow diagram of another data synchronization method provided by the embodiment of the present application;

FIG. 4 is a schematic flowchart of a data synchronization method provided by another embodiment of the present application;

FIG. 5 is a schematic flowchart of another data synchronization method provided by another embodiment of the present application;

FIG. 6 is a schematic flowchart of another data synchronization method provided by another embodiment of the present application;

FIG. 7 is a schematic flowchart of a data synchronization method provided in another embodiment of the present application;

FIG. 8 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of an implementation flow of an alarm prompt provided by an embodiment of the present application.

FIG. 10 is a schematic structural diagram of a data synchronization system provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the application clearer, the application will be further described in detail below in conjunction with the accompanying drawings. All other embodiments obtained under the premise of creative labor belong to the scope of protection of this application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

The embodiment of the present application provides a data synchronization method, as shown in FIG. 1, the method is applied to the first storage node, and the method includes the following steps:

Step 101: If a target operation statement belonging to the target operation type is detected for the target data table, determine at least one second storage node that has an association relationship with the target data table.

Among them, the target operation statement is used to change the content in the target data table.

In the embodiment of the present application, the first storage node is a distributed storage node where the user modifies the content of the target data table stored therein. The target operation type is mainly an operation type used to modify the content in the data table. The at least one second storage node that has an association relationship with the target data table refers to a storage node that stores the target data table.

Step 102. Determine the data synchronization mode of each second storage node.

In this embodiment of the present application, different data synchronization modes are configured for different second storage nodes. During the synchronization process, the data synchronization mode of each second storage node may be adjusted according to actual synchronization conditions. Data synchronization methods include at least real-time synchronization methods and asynchronous synchronization methods.

Step 103, based on each data synchronization mode, send the target operation statement to the corresponding second storage node.

Wherein, based on each data synchronization mode, the target operation statement is sent to the corresponding second storage node, so as to realize the synchronous update of the target data table in the first storage node and the second storage node.

In this embodiment of the application, after the first storage node determines the data synchronization mode of each second storage node, based on the data synchronization mode of each second storage node, the target operation statement is sent to the corresponding second storage node, so that After the second storage node obtains the target operation statement, it can directly execute the target operation statement, and then adjust and update the corresponding data content of the target data table in the second storage node.

Based on the foregoing embodiments, embodiments of the present application provide a data synchronization method, which is applied to a first storage node, and the method includes the following steps:

Step 201: If a target operation statement belonging to the target operation type is detected for the target data table, determine at least one second storage node that has an association relationship with the target data table.

In this embodiment of the application, the target operation statement is that when the first storage node performs a corresponding operation instruction on the target data table of the first storage node from the user, the data service layer of the first storage node responds to the operation instruction and the operation statement belongs to the target operation A statement of type that can be responded to directly by the database in the storage node. The target operation type includes at least the following types: insert (Insert), delete (Delete) and change (Update), and it is determined that the obtained target operation statement belongs to at least one of the above target operation types. When determining at least one second storage node that has an association relationship with the target operation table, it may be determined from the relationship between the pre-stored data table and the storage node. For example, after the target data table is determined, it may be determined to obtain For the identification information of the target data table, at least one second storage node can be obtained by determining the storage node having the identification information of the target data table from the pre-stored relationship between the data table and the storage node. The target operation statement can be, for example, a structured query language (Structured Query Language, SQL) type statement that can directly perform response operations on the database in the storage node, for example, it can be an Insert SQL statement for the target data table, or a Delete statement for the target data table. SQL statements and Update statements for the target data table, etc.

Step 202. Determine the data synchronization mode of each second storage node.

In the embodiment of the present application, the configuration parameters of each second storage node are determined to obtain the data synchronization mode of each second storage node, and then the data synchronization mode of at least one second storage node can be obtained.

Step 203, based on each data synchronization mode, send the target operation statement to the corresponding second storage node.

Wherein, the target operation statement is used to update the target data table in the corresponding second storage node, so as to implement synchronous update of the target data table in the first storage node and the second storage node.

In this embodiment of the application, when the data synchronization mode of the second storage node is real-time synchronization mode, the target operation statement can be sent to the corresponding second storage node in real-time synchronization mode, and the data synchronization mode of the second storage node is asynchronous In the synchronous mode, the target operation statement may be sent to the second storage node in an asynchronous and synchronous manner according to a preset asynchronous time period.

Based on the foregoing embodiments, in other embodiments of the present application, step 203 may be implemented by steps 203a-203c:

Step 203a, if each data synchronization mode is a real-time synchronization mode, determine the table creation statement corresponding to the target data table.

In the embodiment of the present application, when the data synchronization mode of the second storage node is the real-time synchronization mode, the node configuration information of the second storage node is read, and the table creation statement corresponding to the target data table is obtained from the node configuration information .

Step 203b: Generate a first request message based on the table creation statement and the target operation statement.

In the embodiment of the present application, the table creation statement and the target operation statement are respectively filled into the request message to obtain the first request message.

Step 203c, sending the first request message to the corresponding second storage node in a real-time synchronous manner.

Wherein, the first request message includes the target operation statement, and the table creation statement included in the first request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message. data sheet.

In the embodiment of the present application, the first request message is sent to the corresponding second storage node in real time in a real-time synchronous manner, so as to realize sending the target operation statement to the second storage node.

Based on the foregoing embodiments, in other embodiments of the present application, referring to FIG. 2 , the first storage node is also used to perform step 204 before performing step 203:

Step 204, determining the target data volume corresponding to the target operation statement.

In the embodiment of the present application, the first storage node determines the amount of data included in the content of the data table corresponding to the target operation statement, and after determining the number of fields included in the target data table, calculates the difference between the amount of data included in the content of the data table corresponding to the target operation statement and The quotient between the number of fields can get the target data volume.

Correspondingly, step 203 can be realized by step 203d:

Step 203d, based on the target data amount and each data synchronization mode, send the target operation statement to the corresponding second storage node.

In this embodiment of the present application, the default data synchronization mode of the second storage node may be a real-time synchronization mode. In different data synchronization methods, depending on the amount of target data, different preprocessing methods can be used to preprocess the target operation statement to improve the implementation efficiency of the data synchronization method and improve the ability of the target operation statement to be sent to the corresponding second storage. Node sending efficiency.

Based on the foregoing embodiments, in other embodiments of the present application, step 203d may be implemented by steps a11-a13:

Step a11, if each data synchronization method is a real-time synchronization method, and the target data volume is less than a preset threshold, determine a table creation statement corresponding to the target data table.

In this embodiment of the application, the preset threshold can be an empirical value obtained from a large number of experiments, or an empirical value set by the user according to actual needs. During use, the preset threshold can also be based on the response of the target operation statement The duration is dynamically adjusted. The preset threshold has no impact on the performance of the implementation process of the data synchronization method.

Step a12, generating a second request message based on the table creation statement and the target operation statement.

Step a13, sending the second request message to the corresponding second storage node in a real-time synchronous manner.

Wherein, the second request message includes the target operation statement, and the table creation statement included in the second request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the second request message. data sheet.

In this way, when the data synchronization mode corresponding to the second storage node is the real-time synchronization mode, if the target data volume of the target operation statement is less than the preset threshold, the target operation statement and the table creation statement are directly generated into the second request message.

Based on the foregoing embodiments, in other embodiments of the present application, the step "determine the table creation statement corresponding to the target data table" in step 203a or step a11 can be realized through steps b11-b17:

Step b11, determining the call information corresponding to the target operation statement.

In this embodiment of the application, the call information is the data information of the data service layer responding to the target operation statement.

Step b12, from the call information, determine the reference field information used to define the structure of the target data table.

In the embodiment of the present application, the corresponding node configuration information is determined from the call information, and the reference field information corresponding to the structure of the target data table can be obtained by parsing the node configuration information.

Step b13, replacing the reference character in the reference field information with the target character to obtain the target field information.

In this embodiment of the application, the target character can be, for example, "*%*" and the blank character "" in the SQL statement, and the corresponding reference characters are "?" and "[\\s]+". In this way, after the replacement process Get the target field information.

Wherein, after the first storage node executes step b13, it can choose to execute steps b14~b16, or step b17, wherein, if the target field information includes table field entity information, choose to execute steps b14~b16, if the target field information does not include table fields Entity information, choose to execute step b17.

Step b14, if the target field information includes table field entity information, traverse the target field information to obtain the target field name.

Step b15, obtaining the reference field type corresponding to the target field name.

In this embodiment of the application, according to the target field name, the corresponding reference field type is obtained from the table field entity information.

Step b16, using the target statement rule to process the reference field type to obtain a table creation statement.

In the embodiment of this application, the target statement rule is the SQL statement format requirement, for example, if the field is a string type, add single quotes (") to the field, if the field is a date type, convert the field to the date stored in the target data table Format and add single quotes ("). If the field is a valid large number with more than 16 digits, use the conversion string method that comes with the large number to convert the field to a string type and add single quotes ("). In this way, you can get the table statement.

Step b17, if the target field information does not include the table field entity information, obtain the statement corresponding to the target operation statement from the target field storage area to obtain the table creation statement.

In this embodiment of the application, the target field storage area may be, for example, JavaBean.

Based on the foregoing embodiments, in other embodiments of the present application, referring to FIG. 3 , after the first storage node performs step 204, it is also used to perform step 205:

Step 205. If each data synchronization method is a real-time synchronization method, and the target data volume is greater than or equal to a preset threshold, update the corresponding data synchronization method to an asynchronous synchronization method.

In this embodiment of the application, when the data synchronization mode of the second storage node is real-time synchronization mode, if the target data volume is greater than or equal to the preset threshold, if the target operation statement is directly sent to the second The storage node will seriously affect the operation performance of the second storage node. Therefore, the data synchronization mode of the second storage node is adjusted and updated to an asynchronous synchronization mode.

Based on the foregoing embodiments, in other embodiments of the present application, step 203 or step 203d can also be implemented by steps c11-c16:

Step c11. If each data synchronization method is an asynchronous synchronization method, and the target data volume is greater than or equal to a preset threshold, generate a target operation file based on the target operation statement.

In this embodiment of the application, when the data synchronization method of the second storage node is asynchronous synchronization method, if the target data volume is greater than or equal to the preset threshold, create a file and put the target operation statement into the created file , to get the target operation file. Wherein, the file can be named using the identification information of the target data table. Further, since the update operation of the target data table may be continuous, in order to distinguish the target operation statements corresponding to different time periods, the identification information of the target data table is used In addition to naming the file, it can also include the current time and a randomly generated number, that is, use the identification information of the target data table, the current time, and the randomly generated number to name the file storing the target operation statement to obtain the target operation file.

If each data synchronization method is an asynchronous synchronization method and the target data volume is less than a preset threshold, the asynchronous synchronization method is used to directly send the target operation statement to the corresponding second storage node. Exemplarily, when the data synchronization method of the second storage node is an asynchronous synchronization method, if the target data volume of the target operation statement is less than the preset threshold, the first storage node sends the target operation statement every preset asynchronous time period to the second storage node, so that the second storage node executes the target operation statement, and implements a synchronous update operation on the target data table and the target data table in the first storage node.

Step c12, sending the target operation file to the file server.

In this embodiment of the present application, the file server is a server between the first storage node and at least one second storage node for storing files.

Step c13, receiving indication information sent by the file server for indicating the storage location of the target operation file.

In the embodiment of the present application, the indication information may be the identity information (Identity Document, ID) and the corresponding hash value (Hash) generated by the file server to identify the storage file of the target operation file after receiving the target operation file . Correspondingly, the ID of the generated storage file used to identify the target operation file and the corresponding Hash can be obtained by determining the uploaded frame rate (Frames Per Second, FPS).

Step c14, storing the indication information.

Step c15, generating an acquisition instruction according to a preset asynchronous time period.

Wherein, the obtaining instruction is used to instruct to obtain the target operation file from the instruction information of the file server, and obtain the target operation statement to execute the corresponding update operation.

In the embodiment of the present application, the preset asynchronous time period may be an empirical value of a time period set according to actual conditions, or may be an empirical value of time obtained from a large number of experiments. The preset asynchronous time period may be, for example, every 5 minutes, every 30 minutes, or 6:00 every day.

Step c16, sending an acquisition instruction to the corresponding second storage node.

In this embodiment of the application, the first storage node sends the acquisition instruction to the corresponding second storage node immediately after generating the acquisition instruction every preset asynchronous time period, so that the second storage node acquires the instruction information from the file server. The target operation file at the indicated storage location, so that the second storage node executes the corresponding target operation statement in the target operation file.

In some application scenarios, after the first storage node receives the indication information, it generates an acquisition instruction, and sends the acquisition instruction to the corresponding second storage node every preset asynchronous time period, realizing the first storage node and at least one second storage node. The asynchronous synchronization method for the target data table between nodes.

It should be noted that steps c11-c16 may also be executed after step 205, that is, after the data synchronization mode corresponding to the second storage node is updated from real-time synchronization mode to asynchronous synchronization mode, continue to select and execute steps c11-c16.

Based on the foregoing embodiments, in other embodiments of the present application, as shown in FIG. 4 , after the first storage node performs step 203, it is also used to perform steps 206-207:

Step 206: Receive the update result sent by the corresponding second storage node.

Wherein, the update result is an operation result of executing the target operation statement by the second storage node.

In this embodiment of the present application, the update result may be sent by the second storage node to the first storage node in the form of a response message.

When the data synchronization mode is the real-time synchronization mode, after the second storage node receives the second request message, it analyzes the second request message to obtain the target operation statement and the table creation statement; Whether there is a target data table corresponding to the target operation statement in the second storage node; if the target data table does not exist in the second storage node, the second storage node executes the parsed table creation statement to create the corresponding target data table, and then The target data table executes the target operation statement to obtain the operation result, that is, the update result; if the target data table exists in the second storage node, the second storage node judges whether the field information in the existing target data table is consistent with the field information in the table creation statement Consistent, where the field information includes at least field name, field type and field length and other information content; if the field information in the target data table is consistent with the field information in the table creation statement, after the second storage node caches the target data table, Execute the target operation statement for the target data table, update the content in the target data table, and obtain the update result, wherein the purpose of caching before executing the target operation statement for the target data table is that other second storage nodes have abnormal execution of the target operation statement In some cases, the cached target data table can be used to implement the rollback operation; if the field information in the target data table is inconsistent with the field information in the table creation statement, the second storage node obtains the target data table in the second storage node. For the table statement, the second storage node fails to execute the table creation statement and the target operation statement based on the target data table in the second storage node, and obtains an update result.

When the data synchronization method is asynchronous synchronization method, if the second storage node receives the target operation statement every preset asynchronous time period, the execution process of the target operation statement by the second storage node is the same as that of the real-time synchronization method The execution process of the target operation statement by the second storage node is the same, which is not specifically limited here.

When the data synchronization method is asynchronous synchronization method, if the second storage node receives the target operation file every preset asynchronous time period, the second storage node analyzes the target operation file, and obtains the target operation file according to The chronological order of the target operation statements, each time a preset number of operation statements are obtained sequentially for execution, the corresponding execution process is consistent with the execution process of the second storage node for the target operation statements in the case of real-time synchronization, and no specific limitation is made here .

Step 207, based on the update result, update the data synchronization mode of the corresponding second storage node.

In the embodiment of the present application, under different data synchronization modes and different update results, the corresponding update modes of the data synchronization mode of the corresponding second storage node are different.

Based on the foregoing embodiments, in other embodiments of the present application, step 207 may be implemented by step 207a or step 207b:

Step 207a, if the data synchronization mode is the real-time synchronization mode, if the update result indicates that the execution of the target operation statement by the second storage node is abnormal, update the data synchronization mode to the asynchronous synchronization mode.

In this embodiment of the application, when the data synchronization mode is real-time synchronization mode, if the update result indicates that the second storage node successfully executes the target operation statement, there is no need to update the data synchronization mode of the corresponding second storage node, and the corresponding The data synchronization mode of the second storage node is a real-time synchronization mode.

Step 207b. If the data synchronization mode is an asynchronous synchronization mode, if the update result indicates that the second storage node executes the target operation statement successfully, update the data synchronization mode to a real-time synchronization mode.

In this embodiment of the application, when the data synchronization method is asynchronous synchronization method, if the update result indicates that the second storage node fails to execute the target operation statement, the data synchronization method will not be updated, and the data synchronization of the corresponding second storage node will continue to be maintained The method is asynchronous and synchronous.

Based on the foregoing embodiments, in other embodiments of the present application, referring to FIG. 5 , after the first storage node performs step 206, it is also used to perform steps 208-209:

Step 208: If the update result indicates that the execution of the target operation statement by the second storage node is abnormal, determine an exception priority corresponding to the update result.

In the embodiment of the present application, when the update result indicates that the second storage node successfully executes the target operation statement, the current synchronization process ends. If the update result indicates that the execution of the target operation statement by the second storage node is abnormal, an exception priority corresponding to the update result is determined according to the content of the update result.

Step 209, perform an alarm operation corresponding to the abnormal priority.

In the embodiment of the present application, the exception priority assumption includes indicating levels from high to low as: first level, second level, . . . , n level. Assuming that the abnormal priority includes three levels in total, the first level represents the highest priority. Correspondingly, the alarm operation can be, for example, establishing a voice communication connection with the communication device of the relevant person in charge, reminding to deal with the abnormal situation in time, and the second level represents the second highest level. Priority, the corresponding alarm operation can be, for example, to establish a communication connection with the communication device of the relevant person in charge, and remind the relevant person in charge to deal with the abnormal situation when he is free by sending a prompt message. The three marks indicate the lowest priority, and the corresponding The alarm operation may be, for example, generating prompt information and recording it for reference by relevant persons in charge.

Based on the foregoing embodiments, in other embodiments of the present application, as shown in FIG. 6 , after the first storage node performs step 206, it is also used to perform steps 210-211:

Step 210, if the update result indicates that the second storage node executes the target operation statement abnormally, determine the second storage node whose update result indicates that the second storage node successfully executes the target operation statement from at least one second storage node, and obtain at least one third Storage nodes.

In the embodiment of the present application, when the update result indicates that the execution of the target operation statement by the second storage node is abnormal, it means that the synchronization operation on the second storage node has not been successfully executed. Therefore, the synchronization of the target data table is not successful. At this time, from at least one second storage node associated with the target data table, determine the second storage node whose update result indicates that the target operation statement is executed successfully, and obtain at least one third storage node.

Step 211, sending a data rollback control instruction to at least one third storage node.

Wherein, the data rollback control instruction is used to instruct the third storage node to roll back to the data before executing the target operation statement.

In the embodiment of the present application, the first storage node sends the data rollback instruction to at least one third storage node, so that at least one third storage node responds to the data rollback instruction and adopts the target data table cached before executing the target operation statement Replace the target data table after executing the target operation statement to realize the data rollback operation.

Based on the foregoing embodiments, in other embodiments of the present application, as shown in FIG. 7 , after the first storage node performs step 206, it is also used to perform steps 212-213:

Step 212 : When the update result sent by each second storage node indicates that the second storage node successfully executes the target operation statement, count the data processing consumption time of each second storage node to obtain at least one data processing consumption time.

Step 213, based on at least one data processing consumption time, update the preset threshold.

In the embodiment of the present application, the first storage node analyzes at least one data processing consumption time to determine whether the current preset threshold is reasonable, and if not, analyzes according to a processing consumption time indicated to determine a new threshold to update the preset Threshold, if reasonable, there is no need to update the preset threshold. In some application scenarios, after obtaining at least one data processing consumption time, you can also send at least one data processing consumption time to the relevant person in charge, so that the relevant person in charge can analyze at least one data processing consumption time and reset the preset threshold. set up.

Based on the foregoing embodiments, in other embodiments of the present application, step 201 may be implemented by steps 201a-201d:

Step 201a, obtaining mapper declaration parameters and parameter type parameters in the data service layer.

Step 201b: Process the mapper declaration parameters and parameter type parameters by using the target statement rules to obtain a reference operation statement.

In the embodiment of the present application, the target statement rule, that is, the SQL statement rule, is used to analyze the mapper declaration parameters and parameter type parameters, and splicing the parsed content to obtain the SQL statement.

Step 201c, if the operation type of the reference operation statement is the target operation type, determine the target operation statement as the reference operation statement.

In the embodiment of the present application, the reference operation statement is analyzed, and if it is determined that the reference operation statement is the target operation type, that is, Insert, Delete or Update, the reference operation statement is determined as the target operation statement.

Step 201d, determining at least one second storage node that has an association relationship with the target data table.

Based on the aforementioned embodiments, this embodiment of the present application provides a schematic structural diagram of a data synchronization system, as shown in FIG. Among them, the main operation database system, the target standby database system 1, the target standby database system 2 and the target standby database system 3 all include: a business logic application layer, a data service layer, an interceptor and a main operation database (Database, DB). Among them, the interceptor includes: a data synchronization module, a synchronization result processing module, a data delivery exception monitoring and feedback processing module, and a threshold setting module. It should be noted that in FIG. 8 only the structural composition of the main operating system library is completely shown, and the structural composition of the target standby database system 1, the target standby database system 2, and the target standby database system 3 is the same as that of the main operating system library. And the main operation library system corresponds to the aforementioned first storage node, the target backup library system 1, the target backup library system 2 and the target backup library system 3 correspond to the aforementioned at least one second storage node, and the first storage node and the second storage node The functions of the roles can be replaced with each other.

Based on the schematic structural diagram of the data synchronization system shown in FIG. 8, the embodiment of the present application provides a data synchronization method, and the specific implementation process is as follows:

After the business logic layer of the main operation database system receives the operation request sent by the user for updating the target data table in the main operation database, the business logic layer sends the operation request to the data service layer, and the corresponding The data operation statement responds to the operation request and realizes the operation on the target data table. The interceptor in the main operation library system intercepts the corresponding data operation statement in the data service layer, analyzes the intercepted data operation statement, and determines the target operation type for the target data table, such as Insert type, Delete type and Update Target action statement of type etc. After the interceptor in the main operation library system determines that the target operation statement is obtained, at least one second storage node that has an association relationship with the target data table is determined as the target standby database system 1, the target standby database system 2, and the target standby database system 3, and Determine the data synchronization methods corresponding to target backup system 1, target backup system 2, and target backup system 3, and use the data synchronization methods corresponding to target backup system 1, target backup system 2, and target backup system 3 to synchronize the target The operation statement implements the synchronization process, and receives the update results sent by the target standby database system 1, the target standby database system 2, and the target standby database system 3. The main operating database system can also synchronize the data of the target standby database system 1, the target standby database system 2 and the target standby database system 3 according to the update results sent by the target standby database system 1, the target standby database system 2 and the target standby database system 3 Update and adjust preset thresholds, etc., so as to realize a complete closed-loop feedback system and ensure an effective synchronization strategy.

Determining at least one second storage node that has an association relationship with the target data table may be determined from a pre-stored relationship between the data table and the storage node. The relationship between the pre-stored data table and the storage node can be determined from the annotations included in the corresponding framework script, for example, it can be determined from the annotations included in the Spring framework's aspect-oriented programming (Aspect Oriented Programming, AOP), The comment identifies the corresponding data table that needs to be delivered and specifies the target backup system for delivery. The relationship between the pre-stored data table and the storage node can be expressed in the form of a list, for example, as shown in Table 1, DcnRouter is the identification information of the target data table, SysName is the system name information of the system with the target data table, DcnNo is the system number information of the system with the target data table, and ServiceId is the corresponding mapping relationship when information is sent between the systems with the target data table.

Table 1

The interceptor in the main operating library system can be, for example, a plugin function provided by a persistence layer framework such as MyBatis. Remote call can be used between the main operating library system and the target standby library system.

The process for the interceptor to obtain the target operation statement can be, for example: the interceptor obtains the parameters in the data operation statement of the data service layer in response to the operation request, and parses out the command type included in the mapper (mapper) statement (MappedStatement) such as SqlCommandTypeSql from the parameters . According to the command type such as SqlCommandTypeSql, the target operation statement of Insert type, Delete type and Update type is determined.

Determining at least one second storage node that has an association relationship with the target data table can also be that the interceptor obtains the class name according to the intercepted mapping statement identifier such as MappedStatementId, maps the class object information according to the class name, and judges the class object information according to the class object information. Whether there are annotations and annotation parameters added to the class, if annotations and annotation parameters are added, it can be determined to obtain at least one second storage node that has an association relationship with the target data table.

The specific process for the interceptor to determine the target operation statements of the Insert type, Delete type, and Update type according to the command type such as SqlCommandTypeSql can be: the interceptor obtains boundSQL for the MappedStatement and parameter type (ParamObject) in the data service layer. According to the two parameters of MappedStatement and ParamObject, it is parsed and spliced into SQL statements of the target operation type that can be directly executed to obtain the target operation statement. Further, according to the amount of data in the parameter list of boundSQL divided by the number of fields in the table structure, the target data amount to be executed in the SQL statement is obtained.

The interceptor parses the node configuration information in the data operation statement that the data service layer responds to the operation request. Since the node configuration information includes each field information that defines the structure of the target data table, each field information in the node configuration information can be traversed , and splicing out the table creation statement of the target data table. The specific implementation process can be: replace all placeholders "?" in the node configuration information with "*%*", replace all blank characters "[\\s]+" with "", and obtain the replaced node configuration information; Determine whether the table field entity class is used as a parameter in the replaced node configuration information; if the table field entity class is used as a parameter in the replaced node configuration information, loop through the parameter list in turn to determine the parameter set and the parameter that belong to the user. Mapping relationship collection; obtain the field name and field type from the parameter collection passed in by the user and the parameter mapping relationship collection; obtain the field type from the corresponding collection according to the obtained field name. The specific implementation process is: if the field is a string type, add single quotes (") to the field, if the field is a date type, convert the field to the date format stored in the target data table and add single quotes ("), if the field is a valid large number with more than 16 digits, pass the large number The built-in conversion string method converts the field into a string type and adds single quotes (") to get the table creation statement; if the replaced node configuration information does not use the table field entity class as a parameter, get it from JavaBean Entity class batch parameters, you can get the table creation statement. Among them, the table creation statement is also a SQL statement.

The data synchronization module of the interceptor determines the data synchronization mode of each target standby database system. Among them, the initial data synchronization mode of each target standby database system defaults to the real-time synchronization mode. The data synchronization module of the interceptor checks whether the target data volume reaches the set preset threshold value. When the data synchronization mode is the real-time synchronization mode, if the target data volume is less than the preset threshold value, the data synchronization mode is kept as the real-time synchronization mode. The target data volume is greater than or equal to the preset threshold, and the data synchronization method is updated from the real-time synchronization method to the asynchronous synchronization method. When the data synchronization method is an asynchronous synchronization method, if the target data volume is less than the preset threshold, determine that the data synchronization method is specifically asynchronous data synchronization mode; if the target data volume is greater than or equal to the preset threshold value, determine that the data synchronization method is specifically asynchronous file synchronization Way. Among them, the data synchronization method can be realized through two parameters of the table name and the data synchronization method when recording. Further, in the process of the asynchronous and synchronous manner, the format shown in Table 2 may be used to implement a specific asynchronous and synchronous manner. In Table 2, the table_name parameter is used to indicate the table name of the target data table; the dcn_no parameter is used to indicate the number of the target backup system that receives the target operation data; the sql parameter is used to record the SQL that needs to be synchronized when the process_type is SQL Script, otherwise it is empty; id_hash is used to record the ID and Hash value of the storage target operation file converted when the target operation file is uploaded to the file server when the process_type is FILE; process_type is used to identify the specific execution type of the asynchronous and synchronous mode, It is identified by FILE representing the asynchronous file synchronization method and SQL representing the asynchronous data synchronization method; create_time is used to represent the current time when the asynchronous synchronization method is generated, and this event can be used as a logical sequence for subsequent synchronization; sys_seq is used to represent the asynchronous synchronization method The corresponding current serial number is used by the operation and maintenance personnel to use the serial number to find the call chain of the entire process after monitoring the abnormality, so as to quickly locate the problem, execute the processing plan, and evaluate the scope of impact.

Table 2

table_name

sql

id_hash

dcn_no

process_type

create_time

sys_seq

Exemplarily, if the data synchronization module of the interceptor determines that the data synchronization mode of the target backup system 1 is a real-time synchronization mode, the interceptor fills the determined target operation statement and the table creation statement into the real-time request message, and sends the real-time The request message is published to the real-time message queue corresponding to the system number corresponding to the target standby database system 1, and the real-time request message is sent to the target standby database system 1 in the form of a queue message. After receiving the real-time request message, the target standby database system 1 performs the following judgments on the real-time request message: judge whether there is a target data table with a table structure corresponding to the table-building statement in the target standby database system 1; if the target standby database system 1 There is no matching target data table in the real-time request message, execute the table creation statement in the real-time request message to create the target data table, and then execute the target operation statement in the real-time request message to obtain the update result of the target operation statement, and The update result is filled in the response message and sent to the main operating database system; if there is a matching target data table in the target standby database system 1, further check whether the field name, field type, and field length in the table creation statement are in the target standby database system 1 The field names, field types, and field lengths of the target data table are consistent; if the field names, field types, and field lengths are all consistent, first cache the target data table in the target standby database system 1, and then execute the target operation statement to obtain Update the result, and fill the update result into the response message and send it to the main operating library system; if at least one of the field name, field type, and field length is inconsistent, obtain the target data table in the target standby database system 1 through a specific statement such as dataSource Table structure information, get the table creation statement of the target data table in the target standby database system 1, and fill in the response report with the table creation statement of the target data table in the target standby database system 1 and the specific reason for the failed execution. file and send it to the main operating library system.

After the synchronization result processing module of the main operation database system interceptor receives the response message sent by the target standby database system 1, if the response message indicates that the target standby database system 1 has successfully executed synchronously (that is, the aforementioned execution of the target operation statement is successful), the target The data synchronization operation of standby database system 1 is completed. If the response message indicates that the target standby database system 1 fails to execute synchronously (that is, the aforementioned execution of the target operand statement is abnormal), first update the data synchronization mode corresponding to the target standby database system to asynchronous synchronization mode; then find out through the system number of the current update result The standby database system subsystem that has been successfully executed before, finally records the target operation statement of this synchronization and the system number record corresponding to the target standby database system 1 according to Table 2, and updates the process_type field in Table 2 to SQL. At the same time, when the response message indicates that the target standby database system 1 fails to execute synchronously, the abnormal information needs to be persisted. The information structure of the persistent abnormal information can be shown in Table 3, and some content identifiers in Table 3 are the same as those in Table 2. I won’t go into details here. The route_sys_name in Table 3 is used to identify the system name of the standby database where the abnormal execution occurs, which is convenient for the operation and maintenance personnel to locate the system log; remark is used to indicate the remark, which is mainly used to record the cause of the execution failure and serve as an alarm Critical log; urgent_type is used to identify the urgent type, where H indicates high priority, L indicates low priority, and D has completed processing.

table 3

table_name

sql

id_hash

dcn_no

route_sys_name

remark

create_time

urgent_type

Among them, if the reason for the synchronization failure is that at least one of the field name, field type, and field length is inconsistent, the interceptor's data delivery exception monitoring and feedback processing module determines that urgent_type is an exception information record with high priority, that is, H, for high priority The urgent_type can trigger voice communication, such as reminding the relevant person in charge, so that the relevant person in charge can confirm whether the inconsistency of the upstream and downstream fields can be adjusted. After the adjustment is consistent, the urgent_type can be changed to D and processed. If the cause of the failure is the failure caused during the synchronization process, the interceptor's data delivery exception monitoring and feedback processing module determines that the urgent_type is an exception information record with low priority L. For the low priority urgent_type, you can send a prompt message such as a text message or voice information to remind the relevant person in charge, so that the relevant person in charge can centrally arrange time for processing, and the relevant person in charge changes urgent_type to D after processing.

On the premise that the synchronization of the target backup system 1 fails, and the synchronization of the target backup system 2 and the target backup system 3 is successful, since the synchronization of the target backup system 1 fails, the data of the interceptor is sent to the exception monitoring and feedback processing module It is also necessary to perform a data rollback operation on the target standby database system 2 and the target standby database system 3. The main operating library system determines the system numbers of target standby database system 2 and target standby database system 3, assembles the SQL regression statement to be executed into a rollback notification message, and publishes it to the data rollback queue for sending to the corresponding target standby database. Database system 2 and target standby database system 3. After the target standby database system 2 and the target standby database system 3 receive the rollback notification message, they will perform corresponding data rollback: for example, when the target operation statement is Insert type, the newly added and successful data will be deleted; if the target operation statement is Update Type, find out the backup data before updating through the update condition, and restore the data; if the target operation statement is the Delete type, find out the backup data before deleting through the condition to restore.

When the data synchronization module of the interceptor determines that the data synchronization mode of the target standby database system 1 is an asynchronous data synchronization mode, the data synchronization module of the interceptor of the main operation library system fills the target operation statement into the asynchronous request message, and sends the asynchronous request The message is published to the asynchronous message queue corresponding to the system number corresponding to the target standby database system 1, and the target operation data is sent to the target standby database system 1 according to the preset asynchronous time period, for example, every 30 minutes. The implementation process after the target backup system 1 receives the asynchronous request message is the same as the processing steps for the target backup system 1 to receive the real-time request message, and will not be described in detail here.

When the data synchronization module of the interceptor determines that the data synchronization method of the target standby database system 1 is an asynchronous file synchronization method, the data synchronization module of the interceptor of the main operating library system first creates a file in the local system, with the name of the target data table, the current Date and random production number to name the created file; write the target operation statement into the created file; upload the file including the target operation statement to the file server, and the file server will return the storage for the file including the target operation statement The unique identifier of the location, that is, for the id_hash field in Table 2 and Table 3. Then the data synchronization module of the interceptor fills the returned unique identifier into the asynchronous request message, and publishes the asynchronous request message to the asynchronous message queue corresponding to the system number corresponding to the target backup system 1, and transfers the target operation data according to the preset Set the asynchronous time period, for example, to send the messages in the asynchronous message queue to the target standby database system 1 every 30 minutes. The processing steps of the text are the same, and will not be described in detail here. The specific implementation process of writing the target operation statement into the created file can be writing all the statements in the target operation statement into the created file, or grouping the operation statements in the target operation statement according to a certain number, Generate at least one file to achieve asynchronous synchronization.

The asynchronous processing result of the asynchronous synchronous mode is basically the same as the result processing of the synchronous mode, the difference is that the data synchronous mode is updated to real-time synchronous update after receiving the return message of successful data synchronization.

The threshold setting module of the main operation library system interceptor is based on Spring’s AOP annotation @CollectionReccateTime(), to collect and execute data synchronization successfully, the time-consuming period between the detection of the user’s operation request and the detection of the successful synchronization and the current synchronization The amount of target data included, data collection and observation of these two values of the three types of data operations, respectively, can obtain the data time-consuming record results shown in Table 4, so that the relevant person in charge can understand the data shown in Table 4 According to the statistical results, it takes time to adjust the preset threshold by observing the different data volumes of different types of operations, and the adjusted preset threshold is sent to the main operation library system through the dynamic configuration item to realize the subsequent synchronization process.

Table 4

op_type

data_count

timeout

Among them, op_type is used to identify the operation type, data_count is used to identify the data volume of the corresponding operation type, and timeout is used to identify the time-consuming of the operation type. Time-consuming corresponding to an operation type = time-consuming parsing request + ∑ (time-consuming data acquisition + time-consuming processing data) + time-consuming assembling and returning results, ∑ (time-consuming data acquisition + time-consuming processing data) corresponds to the operation type The amount of data and the time-consuming operation statement.

In the data delivery exception monitoring and feedback processing module in the main operation library system, the process of realizing the abnormal alarm can be realized based on the text information platform and the virtual phone outbound platform. Referring to Figure 9, each new emergency type For high-priority abnormal information, the relevant person in charge can be reminded by immediately triggering a phone call reminder. If the relevant person in charge handles the high-priority abnormal information, change the emergency type in the abnormal information to processed. If the relevant person in charge The high-priority exception information is not processed, and the emergency type in the exception record information is kept as high priority, and the trigger logic is put into the delay queue, and the virtual phone outbound platform is triggered again after a certain delay, for example, 5 minutes. Realize the telephone reminder method to remind the relevant person in charge until the emergency type in the exception information record is changed to processed. For records with an urgent type of low priority, an abnormal information scanning operation can be triggered after the data synchronization is completed, that is, a regular polling method is used to scan the records with an urgent type of low priority in the abnormal information records , converting the scanned exception information records with the urgent type of low priority into a request message format and sending them to the text information platform to realize the text information reminder until the urgent type in the abnormal information records is changed to processed. Wherein, the text information platform may be, for example, a short message platform.

For descriptions of the same steps and content in this embodiment as in other embodiments, reference may be made to the descriptions in other embodiments, and details are not repeated here.

Based on the foregoing embodiments, the embodiments of the present application provide a data synchronization system, which can be applied to the data synchronization methods provided in the embodiments corresponding to FIGS. 1 to 7. Referring to FIG. 10, the data synchronization system 3 It may include: a first storage node 31 and at least one second storage node 32, wherein:

The first storage node 31 is used to store the target data table, and execute the data synchronization program to achieve the following steps: if a target operation statement belonging to the target operation type for the target data table is detected, determine the associated relationship with the target data table At least one second storage node; wherein, the target operation statement is used to change the content in the target data table; determine the data synchronization mode of each second storage node; based on each data synchronization mode, send the target operation statement to the corresponding second A storage node, so as to realize the synchronous update of the target data table in the first storage node and the second storage node;

The second storage node 32 is configured to receive the target operation statement; use the target operation statement to perform an update operation on the target data table corresponding to the target operation statement in the second storage node to obtain an update result.

In other embodiments of the present application, when the first storage node 31 executes the step based on each data synchronization method, when sending the target operation statement to the corresponding second storage node, it can be implemented through the following steps:

If each data synchronization method is a real-time synchronization method, determine the table creation statement corresponding to the target data table;

Generate a first request message based on the table creation statement and the target operation statement;

Send the first request message to the corresponding second storage node in a real-time synchronous manner; wherein, the first request message includes a target operation statement, and the table creation statement included in the first request message is used for receiving at the second storage node After the first request message, when it is detected that the target data table does not exist, the target data table is generated.

In other embodiments of the present application, the execution step of the first storage node 31 is also used to perform the following steps before sending the target operation statement to the corresponding second storage node based on each data synchronization method:

Determine the target data volume corresponding to the target operation statement;

Correspondingly, based on each data synchronization mode, send the target operation statement to the corresponding second storage node, including:

Based on the target data amount and each data synchronization mode, the target operation statement is sent to the corresponding second storage node.

In other embodiments of the present application, when the first storage node 31 performs the step of sending the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization method, it can be implemented through the following steps:

If each data synchronization method is a real-time synchronization method, and the target data volume is less than a preset threshold, determine the table creation statement corresponding to the target data table;

Generate a second request message based on the table creation statement and the target operation statement;

Send the second request message to the corresponding second storage node in a real-time synchronous manner; wherein, the second request message includes a target operation statement, and the table creation statement included in the second request message is used to receive the second request message at the second storage node After the second request message, when it is detected that the target data table does not exist, the target data table is generated.

In other embodiments of the present application, when the first storage node 31 performs the step of determining the table creation statement corresponding to the target data table, it may be implemented through the following steps:

Determine the call information corresponding to the target operation statement;

From the call information, determine reference field information for defining the structure of the target data table;

Using the target character to replace the reference character in the reference field information to obtain the target field information;

If the target field information includes table field entity information, traverse the target field information to get the target field name;

Obtain the reference field type corresponding to the target field name;

Use the target statement rule to process the reference field type to obtain the table creation statement;

If the target field information does not include the table field entity information, the statement corresponding to the target operation statement is obtained from the target field storage area to obtain the table creation statement.

In other embodiments of the present application, the first storage node 31 is further configured to perform the following steps:

If each data synchronization method is a real-time synchronization method, and the target data volume is greater than or equal to a preset threshold, update the corresponding data synchronization method to an asynchronous synchronization method.

If each data synchronization method is an asynchronous synchronization method, and the target data volume is greater than or equal to a preset threshold, generate a target operation file based on the target operation statement;

Send the target operation file to the file server;

receiving instruction information sent by the file server for indicating the storage location of the target operation file;

store instructions;

Generate an acquisition instruction according to the preset asynchronous time period; wherein, the acquisition instruction is used to instruct to acquire the target operation file from the instruction information of the file server, and obtain the target operation statement to perform the corresponding update operation;

Sending the acquisition instruction to the corresponding second storage node.

In other embodiments of the present application, the execution step of the first storage node 31 is based on each data synchronization method, and after sending the target operation statement to the corresponding second storage node, it is also used to perform the following steps:

receiving the update result sent by the corresponding second storage node; wherein, the update result is the operation result of the second storage node executing the target operation statement;

Based on the update result, the data synchronization mode of the corresponding second storage node is updated.

In other embodiments of the present application, when the first storage node 31 executes the step of updating the data synchronization mode of the corresponding second storage node based on the update result, it can be implemented through the following steps:

When the data synchronization method is real-time synchronization method, if the update result indicates that the target operation statement executed by the second storage node is abnormal, the update data synchronization method is an asynchronous synchronization method;

When the data synchronization method is an asynchronous synchronization method, if the update result indicates that the second storage node successfully executes the target operation statement, the updated data synchronization method is a real-time synchronization method.

In other embodiments of the present application, after the first storage node 31 performs the step of receiving the update result sent by the corresponding second storage node, it is also used to perform the following steps:

If the update result indicates that the execution of the target operation statement by the second storage node is abnormal, determine the exception priority corresponding to the update result;

Execute the alarm operation corresponding to the abnormal priority.

If the update result indicates that the second storage node executes the target operation statement abnormally, from at least one second storage node, determine the second storage node whose update result indicates that the second storage node successfully executes the target operation statement, and obtain at least one third storage node;

Sending a data rollback control command to at least one third storage node; wherein, the data rollback control command is used to instruct the third storage node to roll back to the data before executing the target operation statement.

In the case that the update result sent by each second storage node indicates that the second storage node successfully executes the target operation statement, counting the data processing consumption time of each second storage node to obtain at least one data processing consumption time;

The preset threshold is updated based on at least one data processing consumption time period.

In other embodiments of the present application, if the first storage node 31 executes the step and detects a target operation statement belonging to the target operation type for the target data table and determines at least one second storage node that has an association relationship with the target data table, it may Do this by following these steps:

Obtain mapper declaration parameters and parameter type parameters in the data service layer;

Process the mapper declaration parameters and parameter type parameters using the target statement rules to obtain the reference operation statement;

If the operation type of the reference operation statement is the target operation type, determine that the target operation statement is the reference operation statement;

At least one second storage node that has an association relationship with the target data table is determined.

It should be noted that, when one or more programs in the embodiments of the present application are executed by the processor, the methods provided in the embodiments corresponding to FIGS. 1 to 7 can be implemented, which will not be repeated here.

Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, referred to as a storage medium for short, where one or more programs are stored in the computer-readable storage medium, and the one or more programs can be used by one or more The processor executes to implement the implementation process of the data synchronization method provided in the embodiments corresponding to FIGS. 1 to 7 , which will not be repeated here.

The above are only examples of the present application, and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Industrial Applicability

An embodiment of the present application provides a data synchronization method, system, and storage medium. The method includes: if a target operation statement belonging to the target operation type is detected for the target data table, determining at least one The second storage node; wherein, the target operation statement is used to change the content in the target data table; determine the data synchronization mode of each of the second storage nodes; based on each of the data synchronization modes, send the The target operation statement is sent to the corresponding second storage node to realize the synchronous update of the target data table in the first storage node and the second storage node, which solves the problem of data synchronization efficiency in the current distributed storage system and effectively guarantees It improves the consistency of data in the distributed storage system and improves the efficiency of data synchronization in the distributed storage system.

Claims

A method for synchronizing data, the method being applied to a first storage node, the method comprising:

If a target operation statement belonging to the target operation type for the target data table is detected, at least one second storage node having an association relationship with the target data table is determined; wherein the target operation statement is used to change the target data table content in

determining a data synchronization mode for each of the second storage nodes;

Based on each of the data synchronization modes, the target operation statement is sent to the corresponding second storage node, so as to realize synchronous update of the target data table in the first storage node and the second storage node.
The method according to claim 1, wherein the sending the target operation statement to the corresponding second storage node based on each of the data synchronization methods includes:

If each of the data synchronization methods is a real-time synchronization method, determine the table creation statement corresponding to the target data table;

Generate a first request message based on the table creation statement and the target operation statement;

Send the first request message to the corresponding second storage node through the real-time synchronization method; wherein, the first request message includes the target operation statement, and the first request message includes The table creation statement is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message.
The method according to claim 1 or 2, wherein, before sending the target operation statement to the corresponding second storage node based on each of the data synchronization methods, the method further comprises:

determining the target data volume corresponding to the target operation statement;

Correspondingly, the sending the target operation statement to the corresponding second storage node based on each data synchronization mode includes:

Based on the target data volume and each of the data synchronization modes, sending the target operation statement to the corresponding second storage node.
The method according to claim 3, wherein the sending the target operation statement to the corresponding second storage node based on the target data volume and each of the data synchronization methods includes:

If each of the data synchronization methods is the real-time synchronization method, and the target data volume is less than a preset threshold, determine the table creation statement corresponding to the target data table;

Generate a second request message based on the table creation statement and the target operation statement;

Send the second request message to the corresponding second storage node through the real-time synchronization method; wherein, the second request message includes the target operation statement, and the second request message includes The table creation statement is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the second request message.
The method according to claim 2 or 4, wherein said determining the table creation statement corresponding to the target data table comprises:

determining the invocation information corresponding to the target operation statement;

From the call information, determine reference field information for defining the structure of the target data table;

replacing the reference character in the reference field information with the target character to obtain the target field information;

If the target field information includes table field entity information, traverse the target field information to obtain the target field name;

Obtain the reference field type corresponding to the target field name;

Processing the reference field type by using target statement rules to obtain the table creation statement;

If the target field information does not include the table field entity information, obtain the statement corresponding to the target operation statement from the target field storage area to obtain the table creation statement.
The method according to claim 4, wherein the method further comprises:

If each of the data synchronization methods is the real-time synchronization method, and the target data volume is greater than or equal to the preset threshold, update the corresponding data synchronization method to be an asynchronous synchronization method.
The method according to claim 2 or 6, wherein the method further comprises:

If each of the data synchronization methods is an asynchronous synchronization method, and the target data volume is greater than or equal to the preset threshold, generate a target operation file based on the target operation statement;

sending the target operation file to a file server;

receiving indication information sent by the file server for indicating the storage location of the target operation file;

storing the instruction information;

Generate an acquisition instruction according to a preset asynchronous time period; wherein, the acquisition instruction is used to instruct to acquire the target operation file from the instruction information of the file server, and obtain the target operation statement to execute the corresponding update operation;

sending the acquiring instruction to the corresponding second storage node.
The method according to claim 1, wherein, after sending the target operation statement to the corresponding second storage node based on each of the data synchronization methods, the method further comprises:

receiving an update result sent by the corresponding second storage node; wherein the update result is an operation result of the second storage node executing the target operation statement;

Based on the update result, update the data synchronization mode of the corresponding second storage node.
The method according to claim 8, wherein, based on the update result, updating the corresponding data synchronization mode of the second storage node comprises:

When the data synchronization method is a real-time synchronization method, if the update result indicates that the execution of the target operation statement by the second storage node is abnormal, update the data synchronization method to an asynchronous synchronization method;

When the data synchronization method is the asynchronous synchronization method, if the update result indicates that the second storage node successfully executes the target operation statement, update the data synchronization method to the real-time synchronization method.
The method according to claim 8, wherein, after receiving the update result sent by the corresponding second storage node, the method further comprises:

If the update result indicates that the execution of the target operation statement by the second storage node is abnormal, determine the exception priority corresponding to the update result;

Execute the alarm operation corresponding to the abnormal priority.
The method according to claim 8, wherein, after receiving the update result sent by the corresponding second storage node, the method further comprises:

If the update result indicates that the execution of the target operation statement by the second storage node is abnormal, determine from at least one of the second storage nodes that the update result indicates that the second storage node successfully executes the target operation statement the second storage node to obtain at least one third storage node;

Sending a data rollback control instruction to the at least one third storage node; wherein the data rollback control instruction is used to instruct the third storage node to roll back to the data before executing the target operation statement.
The method according to claim 8, wherein the method further comprises:

In the case that the update result sent by each of the second storage nodes indicates that the second storage node successfully executes the target operation statement, the data processing consumption time of each of the second storage nodes is counted to obtain at least The duration of the data processing;

The preset threshold is updated based on at least one of the data processing consumption time.
The method according to claim 1, wherein if the target operation statement belonging to the target operation type for the target data table is detected, determining at least one second storage node having an association relationship with the target data table includes:

Obtain mapper declaration parameters and parameter type parameters in the data service layer;

Process the mapper declaration parameters and parameter type parameters by using the target statement rules to obtain the reference operation statement;

If the operation type of the reference operation statement is the target operation type, determine that the target operation statement is the reference operation statement;

Determine at least one second storage node that has an association relationship with the target data table.
A data synchronization system, the system comprising: a first storage node and at least one second storage node; wherein:

The first storage node is configured to determine at least one second storage node associated with the target data table if a target operation statement belonging to the target operation type is detected for the target data table; wherein the target operation The statement is used to change the content in the target data table; determine the data synchronization mode of each of the second storage nodes; based on each of the data synchronization modes, send the target operation statement to the corresponding second storage node node;

The second storage node is configured to receive the target operation statement; use the target operation statement to perform an update operation on the target data table corresponding to the target operation statement in the second storage node, to obtain an updated result.
A storage medium, on which a data synchronization program is stored, and when the data synchronization program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 13 are realized.