CN112163038A - Cross-cluster data synchronization method, device, equipment and storage medium - Google Patents
Cross-cluster data synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112163038A CN112163038A CN202010986063.4A CN202010986063A CN112163038A CN 112163038 A CN112163038 A CN 112163038A CN 202010986063 A CN202010986063 A CN 202010986063A CN 112163038 A CN112163038 A CN 112163038A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- synchronized
- target
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000001360 synchronised effect Effects 0.000 claims abstract description 150
- 238000001914 filtration Methods 0.000 claims abstract description 35
- 238000004590 computer program Methods 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
Abstract
The embodiment of the application discloses a cross-cluster data synchronization method, a cross-cluster data synchronization device, cross-cluster data synchronization equipment and a storage medium. The method comprises the following steps: synchronizing data to be synchronized in the first main node through the data synchronization slave node; according to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data; and sending the target synchronization data to the second main node to realize data synchronization. According to the scheme, data synchronization among cluster nodes can be quickly realized, the problem of data identification conflict in the nodes is solved through data filtering, processing can be performed through a synchronization mechanism of a database, the existing codes do not need to be greatly changed, the complexity of data synchronization is reduced, and the efficiency and the real-time performance of data synchronization are improved.
Description
Technical Field
The embodiment of the application relates to the field of cross-cluster data synchronization, in particular to a cross-cluster data synchronization method, a cross-cluster data synchronization device, cross-cluster data synchronization equipment and a storage medium.
Background
With the continuous expansion of the business scale, the traditional MySQL database based on a single cluster is often difficult to meet the requirements during operation, so a multi-cluster solution of the sub-database is generated at present. After the existing service library is subjected to library division processing according to a certain rule, the operation pressure is obviously reduced. However, when summarizing and processing databases of multiple clusters, how to ensure accuracy and timeliness of data synchronous transmission under the condition of minimally modifying the existing service function is a problem to be solved urgently.
For the case of full data loading, the tables in the target cluster need to be cleaned regularly, and at this time, the service operation needs to be interrupted. For the case of incremental data loading, the data processing logic of the service library where the original cluster is located needs to be modified, changed timestamps are recorded for all DML operations of the table to meet the requirement of timing incremental synchronization, the code change amount is large, and timeliness cannot be met.
Disclosure of Invention
The embodiment of the application provides a cross-cluster data synchronization method, a cross-cluster data synchronization device, a cross-cluster data synchronization equipment and a storage medium, so that real-time synchronization of data is realized without greatly changing codes.
In one embodiment, an embodiment of the present application provides a cross-cluster data synchronization method, where the method includes:
synchronizing data to be synchronized in the first main node through the data synchronization slave node;
according to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data;
and sending the target synchronization data to the second main node to realize data synchronization.
In another embodiment, an embodiment of the present application further provides a cross-cluster data synchronization apparatus, where the apparatus includes:
the data synchronization system comprises a data to be synchronized acquisition module, a data synchronization slave node and a data synchronization module, wherein the data to be synchronized acquisition module is used for synchronizing data to be synchronized in a first master node through a data synchronization slave node;
the target synchronous data determining module is used for filtering the data to be synchronized according to the synchronous request data of the second main node to obtain target synchronous data;
and the target synchronous data sending module is used for sending the target synchronous data to the second main node so as to realize data synchronization.
In another embodiment, an embodiment of the present application further provides a cross-cluster data synchronization device, including: one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-cluster data synchronization method of any one of the embodiments of the present application.
In yet another embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the cross-cluster data synchronization method according to any one of the embodiments of the present application.
In the embodiment of the application, the data to be synchronized in the first master node is synchronized through the data synchronization slave node, so that code change is not needed, and the acquisition of the data to be synchronized can be realized only through the set data synchronization slave node and a master-slave synchronization mechanism based on bottom codes, so that the real-time synchronization of the data is ensured. According to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data; and sending the target synchronization data to the second main node to realize data synchronization, thereby avoiding the problem of data conflict between the first main node and the second main node and ensuring the purity of the data.
Drawings
Fig. 1 is a flowchart of a cross-cluster data synchronization method according to an embodiment of the present application;
fig. 2 is a diagram of a cluster structure according to an embodiment of the present application;
FIG. 3 is a flowchart of a cross-cluster data synchronization method according to another embodiment of the present application;
fig. 4 is a schematic structural diagram of a cross-cluster data synchronization apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a cross-cluster data synchronization device according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Fig. 1 is a flowchart of a cross-cluster data synchronization method according to an embodiment of the present application. The cross-cluster data synchronization method provided by the embodiment can be applied to the data synchronization between the cross-cluster nodes. The method may be specifically executed by a cross-cluster data synchronization apparatus, which may be implemented by software and/or hardware, and may be integrated in a cross-cluster data synchronization device. Referring to fig. 1, the method of the embodiment of the present application specifically includes:
and S110, synchronizing the data to be synchronized in the first main node through the data synchronization slave node.
The method and the device for synchronizing the data between the nodes of the cross-cluster are applicable to data one-way synchronization between the nodes of the cross-cluster. The first host node is located in a first cluster and the second host node is located in a second cluster. A first master node, a slave node, and a data synchronization slave node are included in a first cluster. The first master node may synchronize data to the slave node and the data synchronization slave node. The data to be synchronized may be the full data in the first master node or the incremental data in the first master node.
Illustratively, while the first master node synchronizes the data to be synchronized to the slave node, the data to be synchronized is also synchronized to the data synchronization slave node. Therefore, the slave node can acquire the data synchronized by the first master node in real time through the data synchronization. As shown in fig. 2, only one first slave node and two second slave nodes are shown in fig. 2, but the number of the first slave nodes and the second slave nodes is not limited, and in fact, one or more first slave nodes and one or more second slave nodes may be provided. When data to be synchronized exists in the first master node, the data to be synchronized is synchronized in the first slave node, meanwhile, the data synchronization slave node and the first slave node have the same authority, the first master node regards the data synchronization slave node as the first slave node, and the data to be synchronized is also synchronized in the data synchronization slave node. Similarly, the second master node may synchronize data to the second slave node.
The method has the advantages that the data to be synchronized from the first master node to the first slave node is acquired in real time through the data synchronization slave node, so that other functional structures and codes do not need to be changed, the data to be synchronized can be acquired only through a master-slave synchronization mechanism of the bottom codes, and the complexity of data synchronization is reduced. And when the data synchronization slave node acquires the data to be synchronized, other service operations are not required to be interrupted, and the data synchronization efficiency is improved.
And S120, filtering the data to be synchronized according to the synchronization request data of the second main node to obtain target synchronization data.
Illustratively, the data to be synchronized and the target synchronization data are transmitted in the form of a binary log. The second host node may not need to synchronize all data in the first host node, or the local data in the second host node may have data that conflicts with the data to be synchronized in the first host node, so that the data to be synchronized may be filtered according to the synchronization request data generated by the second host node, thereby obtaining the target synchronization data required by the second host node.
In this embodiment of the present application, before filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data, the method further includes: and setting at least one filter for filtering the data to be synchronized.
As shown in fig. 2, a filter is provided, and receives data to be synchronized, which is synchronized from the first master node by the data synchronization slave node, and filters the data to be synchronized. The number of the filters may also be multiple, and when data of the first master node needs to be synchronized to multiple second master nodes, multiple filters may be set, the filtering rules for the second master nodes are respectively determined, and then the data to be synchronized is filtered to obtain target synchronization data corresponding to the second master nodes.
The beneficial effects of the above scheme are that the data to be synchronized is filtered through the filter, so that the target synchronization data required by the second main node is obtained, the problem of data identification conflict when the data between the two cluster main nodes are synchronized is avoided, and the accuracy and the purity of the data are ensured.
S130, the target synchronization data are sent to the second main node to achieve data synchronization.
Illustratively, the target synchronization data is obtained after the data to be synchronized is filtered, and then the target synchronization data is sent to the second host node, so that data synchronization between the first host node and the second host node is realized. The second master node may send the target synchronization data to the second slave node, thereby implementing master-slave synchronization.
In the embodiment of the application, the data to be synchronized in the first master node is synchronized through the data synchronization slave node, so that code change is not needed, and the acquisition of the data to be synchronized can be realized only through the set data synchronization slave node and a master-slave synchronization mechanism based on bottom codes, so that the real-time synchronization of the data is ensured. According to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data; and sending the target synchronization data to the second main node to realize data synchronization, thereby avoiding the problem of data conflict between the first main node and the second main node and ensuring the purity of the data.
Fig. 3 is a flowchart of a cross-cluster data synchronization method according to another embodiment of the present application. In the embodiment of the present application, details that are not described in detail in the embodiment are referred to in the above embodiment in order to optimize the embodiment on the basis of the above embodiment. Referring to fig. 3, the cross-cluster data synchronization method provided in this embodiment may include:
s210, aiming at the first master node, creating a data synchronization slave node based on a cluster database master-slave synchronization rule.
Illustratively, for a first master node, a data synchronization slave node is created based on master-slave synchronization rules. Because the data synchronization slave node is created based on the master-slave synchronization rule, when the first master node synchronizes the data to be synchronized to the first slave node, the data to be synchronized can be synchronized to the data synchronization slave node based on the master-slave synchronization mechanism used in the bottom layer of the cluster database, so that the synchronization of the data to be synchronized can be realized without changing other functional structures.
The data synchronization method has the advantages that the created data synchronization slave node has the same status as the first slave node, so that the data to be synchronized is synchronized to the data synchronization slave node with a master-slave synchronization mechanism while the first master node synchronizes the data to be synchronized to the first slave node, so that the data synchronization slave node realizes the synchronization of the data in the first master node, other functional structures and codes do not need to be changed, the complexity of operation is reduced, and the efficiency of data synchronization is improved.
And S220, if the first master node synchronizes the data to be synchronized to the first slave node, synchronizing the data to be synchronized in the first master node through the data synchronization slave node.
Specifically, when the first master node synchronizes the data to be synchronized with the first master node, the synchronization process of the data synchronization slave node on the data to be synchronized in the first master node is completed. At this time, the first master node regards the data synchronization slave node as a slave node, so that the data to be synchronized can be synchronized to the data synchronization slave node based on a master-slave data synchronization mechanism of a database used at the bottom layer, and thus, fast and efficient data synchronization is realized.
And S230, determining the identification of the target synchronous data according to the synchronous request data.
And the synchronization request data is generated by the second main node according to the identification of the required synchronization data or the local data identification. For example, the second master node may determine data that needs to be synchronized, and generate synchronization request data according to the data that needs to be synchronized. The local data identification of the second master node can be determined, and the data identification different from the local data identification is used as the identification of the target synchronization data, so that the target synchronization data is synchronized from the first master node in a targeted manner.
S240, filtering the data to be synchronized according to the identification of the target synchronization data to obtain the target synchronization data.
Illustratively, in the data to be synchronized, only the data with the identification of the target synchronization data is reserved as the target synchronization data, and the data with other identifications is filtered and not sent to the second host node, so that the data required to be synchronized by the second host node is determined in a targeted manner, the problem of data pollution caused by data identification conflict and repetition can be avoided, and the purity of the data is improved.
And S250, sending the target synchronization data to the second main node to realize data synchronization.
According to the technical scheme of the embodiment of the application, the synchronization of the data to be synchronized in the first main node is realized by creating the data synchronization slave node and based on the local master-slave synchronization mechanism of the database, so that the synchronization of the data to be synchronized is realized under the condition that other functional structures and codes are not changed, the processing complexity is reduced, and the data synchronization efficiency is improved. In addition, the identification of the target synchronous data is determined, and the data to be synchronized is filtered according to the identification of the target data, so that the problem of data pollution caused by data synchronization due to conflict and repetition of data identification is avoided, and the purity of the data is improved.
In this embodiment of the present application, before filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data, the method further includes: and determining a first offset and a first self-increment step length of the data identifier to be synchronized of the first main node. According to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data, including: and determining a target offset and a target self-increment step length of target synchronous data according to the first offset and the first self-increment step length of the data identifier to be synchronized of the first main node and the synchronous request data.
For example, assuming that the first offset of the data identifier to be synchronized by the first master node is X1, and the first self-increment step size is L1, it is determined that the data identifier whose identification needs to be synchronized by the second master node includes the first offset of X1 and the first self-increment step size is L1 according to the synchronization request data, and therefore, it is determined that the target offset of the target synchronization data is X1, and the target self-increment step size of the target synchronization data is L1. And if the data identifier needing to be synchronized is determined to be contained in the data identifier with the first offset of X1 and the first self-increment step size of L1 according to the synchronization request data, the first offset is X1, and the data identifier with the first self-increment step size of L1 also comprises identifiers except the data identifier needing to be synchronized determined by the synchronization request data, determining a target offset and a target self-increment step size of the data identifier needing to be synchronized according to the synchronization request data. By the scheme, the target offset and the target self-increment step length of the screened target synchronous data needing to be filtered can be accurately determined, so that the filtering process is quicker and more efficient, has pertinence, and improves the filtering efficiency and accuracy.
In this embodiment of the present application, the filtering the data to be synchronized according to the synchronization request data of the second master node to obtain target synchronization data includes: if the local data identifier of the second main node is a second offset and a second self-increment step length, determining a target offset and a target self-increment step length of target data as the first offset and the first self-increment step length; and filtering the data to be synchronized according to the first offset and the first self-increment step length to obtain target synchronization data.
For example, the first offset of the to-be-synchronized data identifier in the first master node is 1, the first self-increment step length is 10, the local data identifier in the second master node is 2, the second self-increment step length is 10, the second master node does not include data identified as 1 and 10, and the data identified as 2 and 10 needs to be filtered out, so that the first offset 1 and the first self-increment step length 10 are used as the target offset and the target self-increment step length of the target synchronization data, the to-be-synchronized data is filtered according to the target offset and the target self-increment step length, only the data identified as 1 in the to-be-synchronized data identifier tail number is reserved, the data identified as 2 in the identifier tail number is filtered out, and collision with the data in the second master node is avoided. In addition, in the process of data loading by the first host node, data except for the identifier of the first offset and the first self-increment step length may be loaded by mistake, so that the first offset and the first self-increment step length are used as the target offset and the target self-increment step length of the target synchronization data, the data to be synchronized is filtered, and the above mentioned cluttered data can be filtered, so that the data synchronized to the second host node is ensured to be pure data, and the conflict of the identifier with the local data of the second host node is avoided.
In this embodiment of the present application, before filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data, the method further includes: setting the identifier of the data to be synchronized generated by the first main node according to a first offset and a first self-increment step length; and setting the identifier of the data to be synchronized generated by the first main node according to a second offset and a second self-increment step length.
For example, when data to be synchronized is loaded in the first master node, the identifier of the data to be synchronized is set according to the first offset being 1 and the first incremental step being 10. When data are loaded in the second host node, the identifier of the loaded data is set according to the second offset as 2, and the second self-increment step length is set as 10, so that the data identifier in the first host node and the data identifier in the second host node are distinguished when the data are loaded, and the data identifier is prevented from colliding when the data in the first host node are synchronized to the second host node, and data pollution is caused.
Fig. 4 is a schematic structural diagram of a cross-cluster data synchronization apparatus according to an embodiment of the present application. The device can be applied to the condition of data synchronization between nodes crossing the cluster. The apparatus may be implemented by software and/or hardware, and the apparatus may be integrated in a cross-cluster data synchronization device. Referring to fig. 4, the apparatus specifically includes:
a to-be-synchronized data acquisition module 310, configured to synchronize to-be-synchronized data in the first master node through the data synchronization slave node;
the target synchronization data determining module 320 is configured to filter the data to be synchronized according to the synchronization request data of the second master node, so as to obtain target synchronization data;
a target synchronization data sending module 330, configured to send the target synchronization data to the second master node, so as to implement data synchronization.
In an embodiment of the present application, the apparatus further includes:
and the creating module is used for creating a data synchronization slave node aiming at the first master node based on a cluster database master-slave synchronization rule.
In this embodiment of the application, the module 310 for acquiring data to be synchronized includes:
and the synchronization unit is used for synchronizing the data to be synchronized in the first master node through the data synchronization slave node if the first master node synchronizes the data to be synchronized to the first slave node.
In an embodiment of the present application, the apparatus further includes:
and the filter setting module is used for setting at least one filter and filtering the data to be synchronized.
In this embodiment of the application, the synchronization request data is generated by the second host node according to the identifier of the required synchronization data or the local data identifier.
In this embodiment of the application, the target synchronization data determining module 320 includes:
the identification determining unit is used for determining the identification of the target synchronous data according to the synchronous request data;
and the filtering unit is used for filtering the data to be synchronized according to the identification of the target synchronization data to obtain the target synchronization data.
In an embodiment of the present application, the apparatus further includes:
and the identification information determining module is used for determining a first offset and a first self-increment step length of the data to be synchronized of the first main node.
In this embodiment of the application, the target synchronization data determining module 320 includes:
and the target identification information determining unit is used for determining the target offset and the target self-increment step length of the target synchronous data according to the first offset and the first self-increment step length of the data identification to be synchronized of the first main node and the synchronous request data.
In this embodiment of the application, the target synchronization data determining module 320 includes:
the information replacement unit is used for determining a target offset and a target self-increment step length of target data as the first offset and the first self-increment step length if the local data of the second main node is identified as a second offset and a second self-increment step length;
and the filtering processing unit is used for filtering the data to be synchronized according to the first offset and the first self-increment step length to obtain target synchronization data.
In an embodiment of the present application, the apparatus further includes:
the first setting module is used for setting the identifier of the data to be synchronized generated by the first main node according to a first offset and a first self-increment step length;
and the second setting module is used for setting the identifier of the data to be synchronized generated by the first main node according to a second offset and a second self-increment step length.
In an embodiment of the present application, the first master node is located in a first cluster, and the second master node is located in a second cluster.
In the embodiment of the application, the data to be synchronized and the target synchronization data are transmitted in the form of a binary log.
The cross-cluster data synchronization device provided by the embodiment of the application can execute the cross-cluster data synchronization method provided by any embodiment of the application, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 5 is a schematic structural diagram of a cross-cluster data synchronization device according to an embodiment of the present application. FIG. 5 illustrates a block diagram of an exemplary cross-cluster data synchronization apparatus 412 suitable for use in implementing embodiments of the present application. The cross-cluster data synchronization apparatus 412 shown in fig. 5 is only an example and should not bring any limitation to the function and the scope of the application of the embodiments.
As shown in FIG. 5, the cross-cluster data synchronization apparatus 412 may include: one or more processors 416; the memory 428 is configured to store one or more programs, when the one or more programs are executed by the one or more processors 416, so that the one or more processors 416 implement the cross-cluster data synchronization method provided in the embodiment of the present application, including:
synchronizing data to be synchronized in the first main node through the data synchronization slave node;
according to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data;
and sending the target synchronization data to the second main node to realize data synchronization.
The components of the cross-cluster data synchronization device 412 may include, but are not limited to: one or more processors or processors 416, a memory 428, and a bus 418 that couples the various device components including the memory 428 and the processors 416.
The cross-cluster data synchronization device 412 typically includes a variety of computer device readable storage media. These storage media may be any available storage media that can be accessed by the cross-cluster data synchronization device 412, including volatile and non-volatile storage media, removable and non-removable storage media.
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in memory 428, such program modules 442 including, but not limited to, an operating device, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 442 generally perform the functions and/or methods of the embodiments described herein.
The cross-cluster data synchronization device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), with one or more devices that enable a user to interact with the cross-cluster data synchronization device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the cross-cluster data synchronization device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, the cross-cluster data synchronization device 412 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) through the network adapter 420. As shown in FIG. 5, network adapter 420 communicates with the other modules of cross-cluster data synchronization apparatus 412 via bus 418. It should be appreciated that, although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with the cross-cluster data synchronization apparatus 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives, and data backup storage devices, among others.
The processor 416 performs various functional applications and data processing by executing at least one of the other programs stored in the memory 428, for example, implementing a cross-cluster data synchronization method provided by the embodiments of the present application.
One embodiment of the present application provides a storage medium containing computer-executable instructions that when executed by a computer processor perform a cross-cluster data synchronization method, comprising:
synchronizing data to be synchronized in the first main node through the data synchronization slave node;
according to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data;
and sending the target synchronization data to the second main node to realize data synchronization.
The computer storage media of the embodiments of the present application may take any combination of one or more computer-readable storage media. The computer readable storage medium may be a computer readable signal storage medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the present application, a computer readable storage medium may be any tangible storage medium that can contain, or store a program for use by or in connection with an instruction execution apparatus, device, or apparatus.
A computer readable signal storage medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal storage medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution apparatus, device, or apparatus.
Program code embodied on a computer readable storage medium may be transmitted using any appropriate storage medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (15)
1. A method for cross-cluster data synchronization, the method comprising:
synchronizing data to be synchronized in the first main node through the data synchronization slave node;
according to the synchronization request data of the second main node, filtering the data to be synchronized to obtain target synchronization data;
and sending the target synchronization data to the second main node to realize data synchronization.
2. The method of claim 1, wherein prior to synchronizing the data to be synchronized in the first master node by the data synchronization slave node, the method further comprises:
and aiming at the first master node, creating a data synchronization slave node based on a cluster database master-slave synchronization rule.
3. The method of claim 1, wherein synchronizing the data to be synchronized in the first master node by the data synchronization slave node comprises:
and if the first master node synchronizes the data to be synchronized to the first slave node, synchronizing the data to be synchronized in the first master node through the data synchronization slave node.
4. The method of claim 1, wherein before filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data, the method further comprises:
and setting at least one filter for filtering the data to be synchronized.
5. The method of claim 1, wherein the synchronization request data is generated by the second master node based on an identification of the desired synchronization data or a local data identification.
6. The method of claim 5, wherein filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data comprises:
determining the identification of target synchronous data according to the synchronous request data;
and filtering the data to be synchronized according to the identification of the target synchronization data to obtain the target synchronization data.
7. The method according to any one of claims 1 to 3, wherein before the data to be synchronized is filtered according to the synchronization request data of the second master node to obtain the target synchronization data, the method further comprises:
and determining a first offset and a first self-increment step length of the data identifier to be synchronized of the first main node.
8. The method of claim 7, wherein filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data comprises:
and determining a target offset and a target self-increment step length of target synchronous data according to the first offset and the first self-increment step length of the data identifier to be synchronized of the first main node and the synchronous request data.
9. The method of claim 7, wherein filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data comprises:
if the local data identifier of the second main node is a second offset and a second self-increment step length, determining a target offset and a target self-increment step length of target data as the first offset and the first self-increment step length;
and filtering the data to be synchronized according to the first offset and the first self-increment step length to obtain target synchronization data.
10. The method of claim 7, wherein before filtering the data to be synchronized according to the synchronization request data of the second master node to obtain the target synchronization data, the method further comprises:
setting the identifier of the data to be synchronized generated by the first main node according to a first offset and a first self-increment step length;
and setting the identifier of the data to be synchronized generated by the first main node according to a second offset and a second self-increment step length.
11. The method of claim 1, wherein the first master node is located in a first cluster and the second master node is located in a second cluster.
12. The method of claim 1, wherein the data to be synchronized and the target synchronization data are transmitted in a binary log.
13. An apparatus for cross-cluster data synchronization, the apparatus comprising:
the data synchronization system comprises a data to be synchronized acquisition module, a data synchronization slave node and a data synchronization module, wherein the data to be synchronized acquisition module is used for synchronizing data to be synchronized in a first master node through a data synchronization slave node;
the target synchronous data determining module is used for filtering the data to be synchronized according to the synchronous request data of the second main node to obtain target synchronous data;
and the target synchronous data sending module is used for sending the target synchronous data to the second main node so as to realize data synchronization.
14. A cross-cluster data synchronization device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the cross-cluster data synchronization method of any of claims 1-12.
15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a cross-cluster data synchronization method according to any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010986063.4A CN112163038A (en) | 2020-09-18 | 2020-09-18 | Cross-cluster data synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010986063.4A CN112163038A (en) | 2020-09-18 | 2020-09-18 | Cross-cluster data synchronization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112163038A true CN112163038A (en) | 2021-01-01 |
Family
ID=73858251
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010986063.4A Pending CN112163038A (en) | 2020-09-18 | 2020-09-18 | Cross-cluster data synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112163038A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113064766A (en) * | 2021-05-07 | 2021-07-02 | 数字广东网络建设有限公司 | Data backup method, device, equipment and storage medium |
| CN113190620A (en) * | 2021-04-30 | 2021-07-30 | 康键信息技术（深圳）有限公司 | Method, device, equipment and storage medium for synchronizing data between Redis clusters |
| CN114567646A (en) * | 2022-03-08 | 2022-05-31 | 京东科技信息技术有限公司 | Data processing method, data processing system, electronic device, and storage medium |
| CN114579671A (en) * | 2022-05-09 | 2022-06-03 | 高伟达软件股份有限公司 | Inter-cluster data synchronization method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8805978B1 (en) * | 2012-02-28 | 2014-08-12 | Symantec Corporation | Distributed cluster reconfiguration |
| CN106372221A (en) * | 2016-09-07 | 2017-02-01 | 华为技术有限公司 | File synchronization method, equipment and system |
-
2020
- 2020-09-18 CN CN202010986063.4A patent/CN112163038A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8805978B1 (en) * | 2012-02-28 | 2014-08-12 | Symantec Corporation | Distributed cluster reconfiguration |
| CN106372221A (en) * | 2016-09-07 | 2017-02-01 | 华为技术有限公司 | File synchronization method, equipment and system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190620A (en) * | 2021-04-30 | 2021-07-30 | 康键信息技术（深圳）有限公司 | Method, device, equipment and storage medium for synchronizing data between Redis clusters |
| CN113190620B (en) * | 2021-04-30 | 2023-10-20 | 康键信息技术（深圳）有限公司 | Method, device, equipment and storage medium for synchronizing data between Redis clusters |
| CN113064766A (en) * | 2021-05-07 | 2021-07-02 | 数字广东网络建设有限公司 | Data backup method, device, equipment and storage medium |
| CN114567646A (en) * | 2022-03-08 | 2022-05-31 | 京东科技信息技术有限公司 | Data processing method, data processing system, electronic device, and storage medium |
| CN114579671A (en) * | 2022-05-09 | 2022-06-03 | 高伟达软件股份有限公司 | Inter-cluster data synchronization method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112163038A (en) | Cross-cluster data synchronization method, device, equipment and storage medium | |
| CN110442560B (en) | Log replay method, device, server and storage medium | |
| CN110647579A (en) | Data synchronization method and device, computer equipment and readable medium | |
| CN106874281B (en) | Method and device for realizing database read-write separation | |
| US9026493B1 (en) | Multi-master RDBMS improvements for distributed computing environment | |
| WO2024041022A1 (en) | Database table alteration method and apparatus, device and storage medium | |
| CN113076304A (en) | Distributed version management method, device and system | |
| CN112241437A (en) | Loop control method, device and equipment for multi-master synchronization of database and storage medium | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| US20130318059A1 (en) | Transfer of data from transactional data sources to partitioned databases in restartable environment | |
| CN112015591A (en) | Log management method, server and database system | |
| CN113761052A (en) | Database synchronization method and device | |
| CN111459882A (en) | Namespace transaction processing method and device of distributed file system | |
| CN114564458B (en) | Method, device, equipment and storage medium for synchronizing data among clusters | |
| CN111061740A (en) | Data synchronization method, equipment and storage medium | |
| CN116049142A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN115080666A (en) | Data synchronization method, system, electronic device and storage medium | |
| WO2022077916A1 (en) | Data processing system, data processing method based on blockchain, and device | |
| CN113760950B (en) | Index data query method, device, electronic equipment and storage medium | |
| EP4170519A1 (en) | Data synchronization method and device | |
| CN114116907A (en) | Database synchronization method and device, electronic equipment and storage medium | |
| CN114490540A (en) | Data storage method, medium, device and computing equipment | |
| CN115470235A (en) | Data processing method, device and equipment | |
| CN112948494A (en) | Data synchronization method and device, electronic equipment and computer readable medium | |
| US10503722B2 (en) | Log management apparatus and log management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |