CN108345617B - Data synchronization method and device and electronic equipment - Google Patents
Data synchronization method and device and electronic equipment Download PDFInfo
- Publication number
- CN108345617B CN108345617B CN201710060057.4A CN201710060057A CN108345617B CN 108345617 B CN108345617 B CN 108345617B CN 201710060057 A CN201710060057 A CN 201710060057A CN 108345617 B CN108345617 B CN 108345617B
- Authority
- CN
- China
- Prior art keywords
- data
- log
- synchronized
- change
- storage space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Abstract
The application discloses a data synchronization method, which comprises the following steps: acquiring sequence identification information of a log corresponding to data to be synchronized; and judging whether the sequence identification information meets the corresponding change condition, if so, applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log. According to the data synchronization method, when data synchronization is carried out, the data change of the data to be synchronized meeting the constraint condition is applied to the data storage space, so that the situation that the data consistency of the data storage space is damaged due to unreliable data transmission is avoided, and the data consistency in the data synchronization process is ensured.
Description
Technical Field
The present application relates to the field of data synchronization technologies, and in particular, to a data synchronization method. The application also relates to a data synchronization device and an electronic device.
Background
With the development of cloud computing, more and more services are required to require a more powerful data center for data storage, and the data center is required to have higher availability, so that the influence of system faults on normal services can be reduced or even eliminated to the greatest extent, therefore, establishment of multiple data centers to achieve service disaster tolerance becomes a necessary choice, and data synchronization among the data centers becomes a crucial link for achieving high availability of services.
At present, many schemes aim at achieving data consistency in a distributed scenario, multiple running copies of a user process are deployed on different servers, and meanwhile, it is guaranteed that only one process serves as a Master node (Master) at the same time, the other processes are Slave nodes (Slave) in a standby state, and once the Master process fails, one Slave is elected as a new Master to serve. However, after the currently elected new Master replaces the failed Master to operate, the data change operation that is not completed before the failed Master fails may arrive after a certain delay due to the influence factors such as data packet loss, retransmission, or network delay, but when the delayed data change operation arrives, the data corresponding to the data change operation is updated by the new Master, and the data change operation that arrives with a delay is executed on the basis, which inevitably damages data consistency.
Disclosure of Invention
The application provides a data synchronization method to solve the defect that the data consistency is damaged in the prior art. The application additionally provides a data synchronization device and an electronic device.
The application provides a data synchronization method, which comprises the following steps:
acquiring sequence identification information of a log corresponding to data to be synchronized;
judging whether the sequence identification information meets the corresponding change condition, if so, executing the next step;
and according to the data change of the data to be synchronized related to the data change operation recorded by the log, applying the data change of the data to be synchronized to a data storage space.
Optionally, the logs are read from the log queue through a master node in the cluster, and when the master node reads the logs from the log queue, the logs are sequentially read from the log queue according to a sequence from first to last according to time when the logs are added into the log queue.
Optionally, the sequence identification information includes at least one of: the sequence number of the log in the log queue is used for acquiring the node identifier of the main node of the log from the log queue and the connection identifier of the data connection for transmitting the log;
wherein the data connection comprises: a TCP connection established between the master node and the data storage space.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is larger than the last change sequence number of the corresponding log when the data change occurs last time, or the sequence number is larger than or equal to the last change sequence number.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
judging whether the sequence number carried in the log is larger than the last sequence number corresponding to the data to be synchronized, if so, executing the next step;
or judging whether the sequence number carried in the log is greater than or equal to the last sequence number corresponding to the data to be synchronized, if so, executing the next step.
Optionally, the data storage space stores data by using a Key/Value data storage model; wherein, Key is the data to be synchronized, and Value is the Value of the data to be synchronized.
Optionally, the last change serial number corresponding to the data to be synchronized is stored through the Key/Value data storage model, and is stored through the Key/Value data storage model in a one-to-one correspondence manner with the data to be synchronized.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is greater than the last log sequence number of the corresponding log when the last data change occurs in the data storage space, or the sequence number is greater than or equal to the last log sequence number.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
judging whether the sequence number carried in the log is larger than the current last log sequence number in the data storage space, if so, executing the next step;
or judging whether the sequence number carried in the log is greater than or equal to the current last log sequence number in the data storage space, and if so, executing the next step.
Optionally, the last log sequence number of the data storage space is stored in the data storage space, and when there is a data change to be applied to the data storage space each time, the last log sequence number of the data storage space is updated according to the sequence number of the log corresponding to the current data change.
Optionally, if the sequence identification information is a node identification of a master node used for acquiring the log from the log queue, the change condition corresponding to the node identification of the master node includes: the node identification is greater than or equal to the node identification of the current master node in the cluster.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
and judging whether the node identification of the main node corresponding to the log is greater than or equal to the node identification of the current main node in the cluster, if so, executing the next step.
Optionally, the node identifier is stored in a preset database, and the database stores the node identifier by using a Key/Value data storage model; wherein, Key is the node identifier, and Value is the Value of the node identifier.
Optionally, the node identifier performs data change in an incremental manner, a new master node is elected in the cluster each time, and the value of the node identifier is incremented by 1.
Optionally, if the sequence identifier information is a connection identifier of a data connection used for transmitting the log, the change condition corresponding to the connection identifier of the data connection includes: the connection identifier is greater than or equal to the connection identifier of the data connection between the current master node and the data storage space in the cluster.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
and judging whether the connection identifier of the data connection for transmitting the log is larger than or equal to the connection identifier of the data connection between the current main node and the data storage space in the cluster, and if so, executing the next step.
Optionally, the connection identifier exists in a preset database, and the database stores the connection identifier by using a Key/Value data storage model; and the Key is the connection identifier, and the Value is the Value of the connection identifier.
Optionally, the connection identifier performs data change in an incremental manner, and when a master node in the cluster creates a new data connection with the data storage space, the value of the connection identifier is incremented by 1.
Optionally, the data change operation includes: data creation operations, data deletion operations, and data modification operations.
Optionally, the cluster includes a master node and at least one slave node, where the master node elects from the nodes of the cluster by invoking a pre-provided distributed lock service, or the master node elects from the nodes of the cluster based on an embedded distributed election protocol; wherein the distributed election protocol comprises: paxos.
Optionally, if the determination result of the step of determining whether the sequence identifier information satisfies the change condition corresponding to the sequence identifier information is that the change condition corresponding to the sequence identifier information is not satisfied, the data change of the data to be synchronized is not allowed to be applied to the data storage space.
Optionally, the data storage space is provided and maintained by a storage system, the step of obtaining sequence identification information of a log corresponding to data to be synchronized, the step of judging whether the sequence identification information meets a change condition corresponding to the sequence identification information, and the step of applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log are performed by the storage system.
The present application further provides a data synchronization apparatus, including:
the sequence identification information acquisition unit is used for acquiring the sequence identification information of the log corresponding to the data to be synchronized;
the sequence identification information judging unit is used for judging whether the sequence identification information meets the corresponding change condition, and if so, the data application unit to be synchronized is operated;
and the data to be synchronized application unit is used for applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
The present application additionally provides an electronic device comprising:
a memory, and a processor;
the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:
acquiring sequence identification information of a log corresponding to data to be synchronized;
and judging whether the sequence identification information meets the corresponding change condition, if so, applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
The application provides a data synchronization method, which comprises the following steps: acquiring sequence identification information of a log corresponding to data to be synchronized; judging whether the sequence identification information meets the corresponding change condition, if so, executing the next step; and according to the data change of the data to be synchronized related to the data change operation recorded by the log, applying the data change of the data to be synchronized to a data storage space.
In the data synchronization method provided by the application, when data synchronization is performed, the sequence identification information of the log corresponding to the data to be synchronized is obtained, and whether the sequence identification information of the log corresponding to the data to be synchronized meets the corresponding change condition is further determined, that is, the method comprises the following steps: judging whether the log corresponding to the data to be synchronized meets the constraint condition of updating the recorded data change operation of the log to the data storage space, if so, applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log corresponding to the data to be synchronized, thereby avoiding the situation that the data consistency of the data storage space is damaged due to unreliable data transmission and ensuring the data consistency in the data synchronization process.
Drawings
FIG. 1 is a process flow diagram of an embodiment of a data synchronization method provided herein;
FIG. 2 is a schematic diagram of a data synchronization interaction method provided in the present application;
FIG. 3 is a schematic diagram of another data synchronization interaction method provided in the present application;
FIG. 4 is a schematic diagram of an embodiment of a data synchronization apparatus provided herein;
fig. 5 is a schematic diagram of an embodiment of an electronic device provided by the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
The application provides a data synchronization method, and additionally provides a data synchronization device and an electronic device. The following detailed description and the description of the steps of the method are individually made with reference to the drawings of the embodiments provided in the present application.
The embodiment of the data synchronization method provided by the application is as follows:
referring to fig. 1, which shows a processing flow chart of an embodiment of a data synchronization method provided by the present application, referring to fig. 2, which shows a schematic diagram of a data synchronization interaction manner provided by the present application, and referring to fig. 3, which shows a schematic diagram of another data synchronization interaction manner provided by the present application.
Step S101, obtaining sequence identification information of a log corresponding to data to be synchronized.
In the embodiment of the application, the cluster consists of at least one node, and the node is a service boundary visible to a user and provides data service for the user; further, several clusters may constitute a cluster set. Meanwhile, a snapshot and log separation architecture is adopted, and under the architecture, each cluster is respectively provided with a data storage space related to each cluster and used for storing respective data snapshots and providing the snapshot function of the total data stored by the data storage space; for each data change operation of a user on a node in a cluster, the data change operation is recorded in a log, and simultaneously, the data change caused by the data change operation is applied to a data storage space of the cluster, all logs are added into a log queue by providing a globally unique log queue, and the data consistency among the clusters is ensured through the log queue: when a new log is submitted in the log queue, each cluster reads the newly submitted log from the log queue and applies the data change related to the data change operation recorded in the read log to the data storage space of each cluster respectively, so that the data consistency among the clusters is ensured.
Specifically, in any one cluster, the logs in the log queue are read by a master node in the cluster, and the data change related to the data change operation recorded in the read logs is applied to the data storage space by the current master node. However, when a master node inside a cluster fails and a new master node is elected, an incomplete data change operation of the failed master node may arrive after the new master node is elected, and in this case, if the incomplete data change operation of the failed master node is applied to the data storage space, it is likely to destroy the consistency of data stored in the data storage space. In the data synchronization method provided by the application, in order to avoid data inconsistency caused by the situation, in the process of applying the data change related to the data change operation recorded in the log to the data storage space, the log which causes the data consistency to be damaged is screened out by judging the operation, and the data change related to the log is not allowed to be applied to the data storage space, so that the problem that the data consistency cannot be met due to the election and replacement of the main node in the cluster is solved.
In the embodiment of the application, the log queue has global uniqueness, all global clusters submit data change operations to the log queue through respective internal nodes, and the currently submitted data change operations are added into the log queue after being converted into logs. Wherein the data change operation comprises: data creation operations, data deletion operations, and data modification operations.
The cluster comprises a master node and at least one slave node, wherein the master node is obtained by electing from the nodes of the cluster by calling a pre-provided distributed lock service, one node is elected from the nodes of the cluster to be used as the master node, and the rest nodes are used as the slave nodes. Meanwhile, when a master node in the cluster fails, one node is selected from the rest of slave nodes as a new master node by calling the distributed lock service, so that the cluster can work normally. In addition, the master node may also elect from among the nodes of the cluster based on an embedded distributed election protocol, for example, a node is elected as the master node among the nodes of the cluster by using the distributed election protocol Paxos, and the remaining nodes are slave nodes.
In specific implementation, the logs are read from the log queue through a main node in the cluster, and when the main node reads the logs from the log queue, the logs are sequentially read from the log queue according to the time of adding the logs into the log queue and the sequence from first to last.
The data to be synchronized in the embodiment of the present application refers to data corresponding to a data change operation submitted to the log queue by a master node in the cluster, and the data is synchronized into the data storage space in a subsequent data synchronization process. The log corresponding to the data to be synchronized refers to a log into which data change operations corresponding to the data to be synchronized are converted after being submitted to the log queue.
The sequence identification information of the log comprises: the sequence number of the log in the log queue is used for acquiring the node identifier of the main node of the log from the log queue and the connection identifier of the data connection for transmitting the log; wherein the data connection comprises: a TCP connection established between the master node and the data storage space. It should be noted that the sequence identification information is not limited to the three data information provided above, and may also be other data information besides the three data information, which is not limited herein.
In a specific implementation, if the sequence identification information of the log is a serial number of the log in the log queue, and the serial number is carried by a corresponding log, correspondingly, this step reads the serial number of the current log contained in the log corresponding to the data to be synchronized. If the sequence identification information of the log is a node identification of a master node for acquiring the log from the log queue, and the node identification or the connection identification is provided by the master node, correspondingly, the node identification or the connection identification provided by the master node is acquired in the step.
Step S102, judging whether the sequence identification information meets the corresponding change condition.
In the step S101, according to the sequence identification information obtained in the step S101, determining whether the sequence identification information satisfies a change condition corresponding to the sequence identification information, and if so, executing a step S103 of applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded in the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space. Two embodiments are specifically illustrated by the following:
the first implementation mode comprises the following steps:
in the present embodiment, the sequence identification information is a sequence number of the log in the log queue, and the change condition corresponding to the sequence number of the log is: the sequence number is larger than the last change sequence number of the corresponding log when the data change occurs last time. Based on this, the step of determining whether the sequence identifier information satisfies the corresponding change condition is implemented in the following manner: judging whether the sequence number carried in the log is larger than the last sequence number corresponding to the data to be synchronized, if so, executing the following step S103, and applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space.
In specific implementation, the data storage space can adopt a Key/Value data storage model for data storage; wherein, Key is the data to be synchronized, and Value is the Value of the data to be synchronized. On this basis, the last change serial number corresponding to the data to be synchronized can be stored through the Key/Value data storage model, and can be stored through the Key/Value data storage model in a one-to-one correspondence manner with the data to be synchronized.
For example, as shown in FIG. 2, there are two nodes in the cluster: frontend 1 and Frontend2, Frontend 1 is Master (Master node), Frontend 1 reads 2 logs from the log queue in turn: log1 and Log2, the sequence id (sequence number, SeqId) in the Log queue is in turn: 1001 and 1002, Log1 and Log2 respectively record 2 times of data modification operations of data X to be synchronized, and sequentially modify the data X to be synchronized into 1 and 2; log1 normally arrives at BulkStorage during front 1 as Master, but front 1 fails during the transfer of Log2, after which front 2 is elected as Master, and front 2 reads 2 logs from the Log queue in turn: log2 (Log 2 execution success response is not received, Log2 needs to be re-acquired) and Log3, where SeqId of Log3 in a Log queue is 1003, data modification operation recorded in Log3 modifies data X to be synchronized into 3, and in addition, time of Log2 read by Frontend 1 reaching BulkStorage due to network delay is later than Log 3; the value of the data to be synchronized X stored in the BulkStorage before execution of Log1 is 0, and the LastProcessSequenceId (last change sequence number, LastSeqId) of the Log Log1 corresponding to the data to be synchronized X is 1000;
(a) when Frontend 1 executes Log1 on BulkStorage, the SeqId of the Log1 corresponding to the data X to be synchronized is 1001, the current LastSeqId of the data X to be synchronized is 1000, whether the SeqId is larger than the LastSeqId is judged through condition updating, and the data X to be synchronized stored in the BulkStorage is modified from 0 to 1 as the value 1001 of the SeqId is larger than the value 1000 of the LastSeqId at the moment; then, the LastSeqId corresponding to the value of the data X to be synchronized being 1 is 1001;
(b) when Frontend2 executes Log2 on BulkStorage, the SeqId of the Log2 corresponding to the data X to be synchronized is 1002, the current LastSeqId of the data X to be synchronized is 1001, whether the SeqId is larger than the LastSeqId is judged through condition updating, and the data X to be synchronized stored in the BulkStorage is modified from 1 to 2 as the value 1002 of the SeqId is larger than the value 1001 of the LastSeqId at the moment; then, the LastSeqId corresponding to the value of the data X to be synchronized being 2 is 1002;
(c) when Frontend2 executes Log3 on BulkStorage, the SeqId of the Log3 corresponding to the data X to be synchronized is 1003, the current LastSeqId of the data X to be synchronized is 1002, whether the SeqId is larger than the LastSeqId is judged through condition updating, and the data X to be synchronized stored in the BulkStorage is modified from 2 to 3 as the value 1003 of the SeqId is larger than the value 1002 of the LastSeqId at the moment; then, the LastSeqId corresponding to the value of the data X to be synchronized being 3 is 1003;
(d) for the Log2 that the delay reaches Frontend2, when Frontend2 executes Log2 on BulkStorage, the SeqId of Log2 corresponding to the data X to be synchronized is 1002, the current LastSeqId of the data X to be synchronized is 1003, and whether the SeqId is greater than the LastSeqId is judged through condition updating, because the value 1002 of the SeqId at this time is less than the value 1002 of the LastSeqId at this time, the value of the data X to be synchronized is not allowed to be updated to the BulkStorage as 2.
Similar to the embodiment provided above, the sequence identification information is a sequence number of the log in the log queue, and the change condition corresponding to the sequence number of the log may be: the sequence number is larger than or equal to the last change sequence number of the corresponding log when the data change occurs last time. Correspondingly, the step of judging whether the sequence identification information meets the corresponding change condition is realized by adopting the following mode: judging whether a serial number carried in the log is larger than or equal to a last serial number corresponding to the data to be synchronized, if so, executing a step S103, and applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space.
In addition, in a specific implementation, the sequence identification information is a sequence number of the log in the log queue, and the change condition corresponding to the sequence number of the log may further be: the sequence number is greater than the last log sequence number of the corresponding log when the last data change occurs in the data storage space, or the sequence number is greater than or equal to the last log sequence number. Based on this, the step of determining whether the sequence identifier information satisfies the corresponding change condition is implemented in the following manner: judging whether the sequence number carried in the log is larger than the current last log sequence number in the data storage space, if so, executing the following step S103, and applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space. Or, judging whether a sequence number carried in the log is greater than or equal to a current last log sequence number in the data storage space, if so, executing the following step S103, and applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space.
On the basis, the last log serial number of the data storage space is stored in the data storage space, and the last log serial number of the data storage space is updated according to the serial number of the log corresponding to the current data change when data change is applied to the data storage space every time.
The second embodiment:
in the present embodiment, the sequence identification information is a node identification of a master node used for acquiring the log from the log queue, and the change condition corresponding to the node identification of the master node is: the node identification is greater than or equal to the node identification of the current master node in the cluster. Based on this, the step of determining whether the sequence identification information satisfies the corresponding change condition is implemented by the following method: judging whether the node identification of the main node corresponding to the log is larger than or equal to the node identification of the current main node in the cluster, namely judging whether the node identification of the main node obtaining the log is larger than or equal to the node identification of the current main node in the cluster, if so, executing the following step S103, and applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space.
In specific implementation, the node identifier can be stored through a preset database, and the database stores the node identifier by adopting a Key/Value data storage model; wherein, Key is the node identifier, and Value is the Value of the node identifier. Specifically, the node identifier performs data change in an incremental manner, a new master node is elected in the cluster each time, and the value of the node identifier is increased by 1.
For example, as shown in FIG. 3, there are two nodes in the cluster: front 1 and front 2, where front 1 is Master, front 1 stores a node identifier (Epoch) in a database as 003, and front 1 sequentially reads 2 logs from a log queue: respectively recording 2 times of data modification operations of data X to be synchronized in Log1 and Log2, Log1 and Log2, and sequentially modifying the data X to be synchronized into 1 and 2; log1 normally reaches BulkStorage during Frontend 1 as Master, but Frontend 1 fails during transmission of Log2, thereafter Frontend2 is elected to be Master, Epoch stored in a database by Frontend2 is 003+1 ═ 004, Frontend2 reads Log2 and Log3 from a Log queue in sequence, data modification operation recorded in Log3 modifies data X to be synchronized into 3, and furthermore, Log2 read by Frontend 1 reaches BulkStorage later than Log3 due to network delay;
(a) when the Log1 is executed on the BulkStorage by the Log1, both the main node corresponding to the Log1 and the current main node in the cluster are Frontend 1, when the Epoch of the Log1 corresponding to the main node Frontend 1 and the Epoch of the current main node Frontend 1 in the cluster are judged through condition updating, the epochs are equal and are 003, and the data X to be synchronized stored in the BulkStorage is modified from 0 to 1;
(b) when the Log2 is executed on the BulkStorage by the Log2, both the main node corresponding to the Log2 and the current main node in the cluster are Frontend2, when the condition updating judges that the Epoch of the Log2 corresponding to the main node Frontend2 and the Epoch of the current main node Frontend2 in the cluster are equal and 004, and the data X to be synchronized stored in the BulkStorage is modified from 1 to 2;
(c) when the Log3 is executed on the BulkStorage by the Log2, both the main node corresponding to the Log3 and the current main node in the cluster are Frontend2, when the condition updating judges that the Epoch of the Log3 corresponding to the main node Frontend2 and the Epoch of the current main node Frontend2 in the cluster are equal and 004, and the data X to be synchronized stored in the BulkStorage is modified from 2 to 3;
(d) for the Log2 that the time delay reaches the front 2, when the front 2 executes the Log2 on the BulkStorage, the Epoch of the main node front 1 corresponding to the Log2 is 003, the Epoch of the current main node front 2 in the cluster is 004, and when the Log2 is judged to correspond to the Epoch of the main node front 1 and the Epoch of the current main node front 2 in the cluster through condition updating, since the value (003) of the Epoch of the front 1 is smaller than the value (004) of the Epoch of the front 2, the value of the data X to be synchronized is not allowed to be updated to the BulkStorage as 2.
In practical applications, it may also occur that a TCP connection between a master node in the cluster and the data storage space is disconnected, and although a process running the master node is not hung, the master node reestablishes the TCP connection with the data storage space, in this case, there is also a possibility that a log read by the master node (a log transmitted through an old TCP connection) arrives at the data storage space with a delay (arrives during transmission of a log through a new TCP connection), and if the log arrived with a delay is executed in the data storage space, the consistency of data in the data storage space may be damaged. For this situation, the following embodiments provided in the present application can ensure the data consistency of the data storage space:
in the present embodiment, the sequence identifier information is a connection identifier of a TCP connection used for transmitting the log, and the change condition corresponding to the connection identifier of the TCP connection is: the connection identification is larger than or equal to the connection identification of the TCP connection between the current main node and the data storage space in the cluster. Based on this, the step of determining whether the sequence identifier information satisfies the corresponding change condition is implemented in the following manner: judging whether the connection identifier of the TCP connection for transmitting the log is larger than or equal to the connection identifier of the TCP connection between the current main node and the data storage space in the cluster, if so, executing the following step S103, and applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log; and if not, not allowing the data change of the data to be synchronized to be applied to the data storage space.
Similar to the node identifier in the foregoing embodiment, the connection identifier may also be stored in a preset database, and a Key/Value data storage model is used to store the connection identifier, where Key is the connection identifier and Value is a Value of the connection identifier. And the connection identifier is subjected to data change in an incremental mode, and when a master node in the cluster establishes a TCP connection with the data storage space, the value of the connection identifier is increased by 1.
Step S103, according to the data change of the data to be synchronized related to the data change operation recorded by the log, applying the data change of the data to be synchronized to a data storage space.
The precondition for this step to be implemented is that the determination result of determining whether the sequence identifier satisfies the corresponding change condition in step S102 is yes, and the data change of the data to be synchronized is applied to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded in the log, which may be specifically referred to as an example in the two embodiments provided in step S102.
In practical applications, the data storage space may be provided and maintained by a storage system, and accordingly, the step of obtaining the sequence identification information of the log corresponding to the data to be synchronized in step S101, the step of determining whether the sequence identification information satisfies the change condition corresponding to the sequence identification information in step S102, and the step of applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded in the log in step S103 are executed by the storage system.
To sum up, in the data synchronization method provided by the present application, when data synchronization is performed, by obtaining the sequence identification information of the log corresponding to the data to be synchronized, and further determining whether the sequence identification information of the log corresponding to the data to be synchronized satisfies the change condition corresponding thereto, that is: judging whether the log corresponding to the data to be synchronized meets the constraint condition of updating the recorded data change operation of the log to the data storage space, if so, applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log corresponding to the data to be synchronized, thereby avoiding the situation that the data consistency of the data storage space is damaged due to unreliable data transmission and ensuring the data consistency in the data synchronization process.
The embodiment of the data synchronization device provided by the application is as follows:
in the foregoing embodiment, a data synchronization method is provided, and correspondingly, a data synchronization apparatus is also provided in the present application, which is described below with reference to the accompanying drawings.
Referring to fig. 4, a schematic diagram of an embodiment of a data synchronization apparatus provided in the present application is shown.
Since the device embodiment corresponds to the method embodiment provided above, please refer to the corresponding description of the method embodiment for reading the content of this embodiment. The device embodiments described below are merely illustrative.
The application provides a data synchronization device, includes:
a sequence identification information obtaining unit 401, configured to obtain sequence identification information of a log corresponding to data to be synchronized;
a sequence identifier determining unit 402, configured to determine whether the sequence identifier satisfies a change condition corresponding to the sequence identifier, and if so, run a to-be-synchronized data applying unit 403;
the data to be synchronized applying unit 403 is configured to apply the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
Optionally, the logs are read from the log queue through a master node in the cluster, and when the master node reads the logs from the log queue, the logs are sequentially read from the log queue according to a sequence from first to last according to time when the logs are added into the log queue.
Optionally, the sequence identification information includes at least one of: the sequence number of the log in the log queue is used for acquiring the node identifier of the main node of the log from the log queue and the connection identifier of the data connection for transmitting the log;
wherein the data connection comprises: a TCP connection established between the master node and the data storage space.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is larger than the last change sequence number of the corresponding log when the data change occurs last time, or the sequence number is larger than or equal to the last change sequence number.
Optionally, the sequence identifier determining unit 402 includes:
a first determining subunit, configured to determine whether a sequence number carried in the log is greater than a last sequence number corresponding to the data to be synchronized, if yes, run the data to be synchronized application unit 403;
a second determining subunit, configured to determine whether a sequence number carried in the log is greater than or equal to a last sequence number corresponding to the data to be synchronized, and if yes, run the data to be synchronized application unit 403.
Optionally, the data storage space stores data by using a Key/Value data storage model; wherein, Key is the data to be synchronized, and Value is the Value of the data to be synchronized.
Optionally, the last change sequence number corresponding to the data to be synchronized is stored through the Key/Value data storage model, and is stored through the Key/Value data storage model in a one-to-one correspondence manner with the data to be synchronized.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is greater than the last log sequence number of the corresponding log when the last data change occurs in the data storage space, or the sequence number is greater than or equal to the last log sequence number.
Optionally, the sequence identifier determining unit 402 includes:
a third determining subunit, configured to determine whether a sequence number carried in the log is greater than a current last log sequence number in the data storage space, and if so, run the to-be-synchronized data application unit 403;
a fourth determining subunit, configured to determine whether a sequence number carried in the log is greater than or equal to a current last log sequence number in the data storage space, and if so, run the to-be-synchronized data application unit 403.
Optionally, the last log sequence number of the data storage space is stored in the data storage space, and when there is a data change to be applied to the data storage space each time, the last log sequence number of the data storage space is updated according to the sequence number of the log corresponding to the current data change.
Optionally, if the sequence identification information is a node identification of a master node used for acquiring the log from the log queue, the change condition corresponding to the node identification of the master node includes: the node identification is greater than or equal to the node identification of the current master node in the cluster.
Optionally, the sequence identifier determining unit 402 includes:
a fifth judging subunit, configured to judge whether the node identifier of the master node corresponding to the log is greater than or equal to the node identifier of the current master node in the cluster, and if so, run the to-be-synchronized data application unit 403.
Optionally, the node identifier is stored in a preset database, and the database stores the node identifier by using a Key/Value data storage model; wherein, Key is the node identifier, and Value is the Value of the node identifier.
Optionally, the node identifier performs data change in an incremental manner, a new master node is elected in the cluster each time, and the value of the node identifier is incremented by 1.
Optionally, if the sequence identifier information is a connection identifier of a data connection used for transmitting the log, the change condition corresponding to the connection identifier of the data connection includes: the connection identifier is greater than or equal to the connection identifier of the data connection between the current master node and the data storage space in the cluster.
Optionally, the sequence identifier determining unit 402 includes:
a sixth determining subunit, configured to determine whether a connection identifier of a data connection for transmitting the log is greater than or equal to a connection identifier of a data connection between the current master node and the data storage space in the cluster, and if so, run the to-be-synchronized data application unit 403.
Optionally, the connection identifier exists in a preset database, and the database stores the connection identifier by using a Key/Value data storage model; wherein, Key is the connection identifier, and Value is the Value of the connection identifier.
Optionally, the connection identifier performs data change in an incremental manner, and when a master node in the cluster creates a new data connection with the data storage space, the value of the connection identifier is incremented by 1.
Optionally, the data change operation includes: data creation operations, data deletion operations, and data modification operations.
Optionally, the cluster includes a master node and at least one slave node, where the master node elects from the nodes of the cluster by invoking a pre-provided distributed lock service, or the master node elects from the nodes of the cluster based on an embedded distributed election protocol;
wherein the distributed election protocol comprises: paxos.
Optionally, if the determination result output by the sequence identifier determining unit 402 is that the sequence identifier does not satisfy the change condition corresponding to the sequence identifier, the data change of the data to be synchronized is not allowed to be applied to the data storage space.
Optionally, the data storage space is provided and maintained by a storage system, and the sequence identifier information obtaining unit 401, the sequence identifier information determining unit 402, and the to-be-synchronized data application unit 403 operate through the storage system.
The embodiment of the electronic equipment provided by the application is as follows:
in the foregoing embodiment, a data synchronization method is provided, and in addition, an electronic device for implementing the data synchronization method is provided in the present application, which is described below with reference to the accompanying drawings.
Referring to fig. 5, a schematic diagram of an electronic device provided in the present embodiment is shown.
The electronic device provided by the present application is used for implementing the data synchronization method provided by the present application, and this embodiment corresponds to the data synchronization method embodiment provided above, and please refer to the corresponding description of the data synchronization method embodiment provided above for reading the content of this embodiment. The embodiments described below are merely illustrative.
The application provides an electronic device, including:
a memory 501, and a processor 502;
the memory 501 is configured to store computer-executable instructions, and the processor 502 is configured to execute the computer-executable instructions to:
acquiring sequence identification information of a log corresponding to data to be synchronized;
and judging whether the sequence identification information meets the corresponding change condition, if so, applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
Optionally, the logs are read from the log queue through a master node in the cluster, and when the master node reads the logs from the log queue, the logs are sequentially read from the log queue according to a sequence from first to last according to time when the logs are added into the log queue.
Optionally, the sequence identification information includes at least one of: the sequence number of the log in the log queue is used for acquiring the node identifier of the main node of the log from the log queue and the connection identifier of the data connection for transmitting the log;
wherein the data connection comprises: a TCP connection established between the master node and the data storage space.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is larger than the last change sequence number of the corresponding log when the data change occurs last time, or the sequence number is larger than or equal to the last change sequence number.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
judging whether a sequence number carried in the log is larger than a last sequence number corresponding to the data to be synchronized, if so, executing data change of the data to be synchronized related to data change operation recorded by the log, and applying the data change of the data to be synchronized to a data storage space instruction;
or judging whether a sequence number carried in the log is larger than or equal to a last sequence number corresponding to the data to be synchronized, if so, executing the data change of the data to be synchronized related to the data change operation recorded by the log, and applying the data change of the data to be synchronized to a data storage space.
Optionally, the data storage space stores data by using a Key/Value data storage model; wherein, Key is the data to be synchronized, and Value is the Value of the data to be synchronized.
Optionally, the last change serial number corresponding to the data to be synchronized is stored through the Key/Value data storage model, and is stored through the Key/Value data storage model in a one-to-one correspondence manner with the data to be synchronized.
Optionally, if the sequence identification information is a sequence number of the log in the log queue, the change condition corresponding to the sequence number of the log includes: the sequence number is greater than the last log sequence number of the corresponding log when the last data change occurs in the data storage space, or the sequence number is greater than or equal to the last log sequence number.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
judging whether a sequence number carried in the log is larger than a current last log sequence number in the data storage space, if so, executing data change of the data to be synchronized related to data change operation recorded by the log, and applying the data change of the data to be synchronized to a data storage space instruction;
or judging whether a serial number carried in the log is greater than or equal to a current last log serial number in the data storage space, if so, executing the data change of the data to be synchronized related to the data change operation recorded by the log, and applying the data change of the data to be synchronized to a command in the data storage space.
Optionally, the last log sequence number of the data storage space is stored in the data storage space, and when a data change is applied to the data storage space each time, the last log sequence number of the data storage space is updated according to the sequence number of the log corresponding to the current data change.
Optionally, if the sequence identification information is a node identification of a master node used for acquiring the log from the log queue, the change condition corresponding to the node identification of the master node includes: the node identification is greater than or equal to the node identification of the current master node in the cluster.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
and judging whether the node identification of the main node corresponding to the log is larger than or equal to the node identification of the current main node in the cluster, if so, executing the data change of the data to be synchronized related to the data change operation recorded by the log, and applying the data change of the data to be synchronized to a data storage space.
Optionally, the node identifier is stored in a preset database, and the database stores the node identifier by using a Key/Value data storage model; wherein, Key is the node identifier, and Value is the Value of the node identifier.
Optionally, the node identifier performs data change in an incremental manner, a new master node is elected in the cluster each time, and the value of the node identifier is incremented by 1.
Optionally, if the sequence identifier information is a connection identifier of a data connection used for transmitting the log, the change condition corresponding to the connection identifier of the data connection includes:
the connection identifier is greater than or equal to the connection identifier of the data connection between the current master node and the data storage space in the cluster.
Optionally, the determining whether the sequence identifier information satisfies the corresponding change condition is implemented by the following method:
and judging whether the connection identifier of the data connection for transmitting the log is larger than or equal to the connection identifier of the data connection between the current main node and the data storage space in the cluster, if so, executing the data change of the data to be synchronized related to the data change operation recorded by the log, and applying the data change of the data to be synchronized to a data storage space.
Optionally, the connection identifier exists in a preset database, and the database stores the connection identifier by using a Key/Value data storage model; wherein, Key is the connection identifier, and Value is the Value of the connection identifier.
Optionally, the connection identifier performs data change in an incremental manner, and when a master node in the cluster creates a new data connection with the data storage space, the value of the connection identifier is incremented by 1.
Optionally, the data change operation includes: data creation operations, data deletion operations, and data modification operations.
Optionally, the cluster includes a master node and at least one slave node, where the master node elects from the nodes of the cluster by invoking a pre-provided distributed lock service, or the master node elects from the nodes of the cluster based on an embedded distributed election protocol; wherein the distributed election protocol comprises: paxos.
Optionally, if the execution result of the change condition instruction that determines whether the sequence identifier information satisfies the corresponding change condition instruction is that the sequence identifier information does not satisfy the change condition corresponding to the sequence identifier information, the data change of the data to be synchronized is not allowed to be applied to the data storage space.
Optionally, the data storage space is provided and maintained by a storage system, the sequence identification information instruction of the log corresponding to the data to be synchronized is obtained, whether the sequence identification information meets the change condition instruction corresponding to the sequence identification information is judged, and the data change of the data to be synchronized is applied to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log, and is executed on the storage system.
Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Claims (11)
1. A method of data synchronization, comprising:
acquiring a serial number of a log in a log queue corresponding to data to be synchronized, acquiring a node identifier of a main node of the log from the log queue, and transmitting a connection identifier of data connection of the log; wherein the data connection comprises: the TCP connection is established between the main node and the data storage space; the data to be synchronized refers to data corresponding to data change operation submitted to the log queue by a main node in the cluster; the log corresponding to the data to be synchronized refers to a log converted after the data change operation corresponding to the data to be synchronized is submitted to a log queue; the log is read from the log queue through a master node in the cluster;
judging whether the serial number is larger than a last-bit change serial number of a corresponding log when data change occurs last time or whether the serial number is larger than or equal to the last-bit change serial number; if yes, executing the next step;
and according to the data change of the data to be synchronized related to the data change operation recorded by the log, applying the data change of the data to be synchronized to a data storage space.
2. The data synchronization method according to claim 1, wherein when the master node reads the logs from the log queue, the master node reads the logs from the log queue in sequence from first to last according to the time when the logs are added to the log queue.
3. The data synchronization method according to claim 1, wherein the data storage space adopts a Key/Value data storage model for data storage;
wherein, Key is the data to be synchronized, and Value is the Value of the data to be synchronized.
4. The data synchronization method according to claim 3, wherein the last change sequence number corresponding to the data to be synchronized is stored through the Key/Value data storage model, and is stored through the Key/Value data storage model in a one-to-one correspondence with the data to be synchronized.
5. The data synchronization method according to claim 1, wherein a last log sequence number of the data storage space is stored in the data storage space, and wherein each time a data change is applied to the data storage space, the last log sequence number of the data storage space is updated according to a sequence number of a log corresponding to the current data change.
6. The data synchronization method of claim 1, wherein the data change operation comprises:
data creation operations, data deletion operations, and data modification operations.
7. The data synchronization method according to claim 1, wherein the cluster comprises a master node and at least one slave node, the master node is obtained by electing from the nodes of the cluster by invoking a pre-provided distributed lock service, or the master node is obtained by electing from the nodes of the cluster based on an embedded distributed election protocol;
wherein the distributed election protocol comprises: paxos.
8. The method according to any of claims 1 to 7, wherein if the sequence number is smaller than the last change sequence number, the data change of the data to be synchronized is not allowed to be applied to the data storage space.
9. The data synchronization method according to any one of claims 1 to 7, wherein the data storage space is provided by a storage system and maintained, the step of obtaining a sequence number of a log in a log queue corresponding to data to be synchronized, the step of determining whether the sequence number satisfies a change condition corresponding to the sequence number, and the step of applying a data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to a data change operation recorded by the log are performed by the storage system.
10. A data synchronization apparatus, comprising:
the device comprises a serial number acquisition unit, a synchronization unit and a synchronization unit, wherein the serial number acquisition unit is used for acquiring a serial number of a log in a queue of the log corresponding to data to be synchronized; the node identification of the main node is used for acquiring the log from the log queue, and the connection identification of the data connection is used for transmitting the log; wherein the data connection comprises: the TCP connection is established between the main node and the data storage space; the data to be synchronized refers to data corresponding to data change operation submitted to the log queue by a main node in the cluster; the log corresponding to the data to be synchronized refers to a log converted after the data change operation corresponding to the data to be synchronized is submitted to a log queue; the log is read from the log queue through a master node in the cluster; a serial number judging unit, configured to judge whether the serial number is greater than a last-order-changed serial number of a corresponding log when data change occurs last time, or whether the serial number is greater than or equal to the last-order-changed serial number, and if yes, run a to-be-synchronized data application unit;
and the data to be synchronized application unit is used for applying the data change of the data to be synchronized to the data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
11. An electronic device, comprising:
a memory, and a processor;
the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:
acquiring a serial number of a log in a log queue corresponding to data to be synchronized, acquiring a node identifier of a main node of the log from the log queue, and transmitting a connection identifier of data connection of the log; wherein the data connection comprises: the TCP connection is established between the main node and the data storage space; the data to be synchronized refers to data corresponding to data change operation submitted to the log queue by a main node in the cluster; the log corresponding to the data to be synchronized refers to a log converted after the data change operation corresponding to the data to be synchronized is submitted to a log queue; the log is read from the log queue through a master node in the cluster;
and judging whether the sequence number is larger than a last change sequence number of a corresponding log when the data change occurs last time, or whether the sequence number is larger than or equal to the last change sequence number, if so, applying the data change of the data to be synchronized to a data storage space according to the data change of the data to be synchronized related to the data change operation recorded by the log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710060057.4A CN108345617B (en) | 2017-01-24 | 2017-01-24 | Data synchronization method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710060057.4A CN108345617B (en) | 2017-01-24 | 2017-01-24 | Data synchronization method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108345617A CN108345617A (en) | 2018-07-31 |
| CN108345617B true CN108345617B (en) | 2022-05-06 |
Family
ID=62962900
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710060057.4A Active CN108345617B (en) | 2017-01-24 | 2017-01-24 | Data synchronization method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108345617B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110851526B (en) * | 2018-08-02 | 2023-04-11 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN109639777B (en) * | 2018-11-28 | 2021-12-10 | 优刻得科技股份有限公司 | Data synchronization method, device and system and non-volatile storage medium |
| CN110297866A (en) * | 2019-05-20 | 2019-10-01 | 平安普惠企业管理有限公司 | Method of data synchronization and data synchronization unit based on log analysis |
| CN112015804A (en) * | 2019-05-28 | 2020-12-01 | 阿里巴巴集团控股有限公司 | Data synchronization method, device, equipment and storage medium |
| CN110399095B (en) * | 2019-06-12 | 2022-12-02 | 厦门网宿有限公司 | Storage space statistical method and device |
| CN112835885B (en) * | 2019-11-22 | 2023-09-01 | 北京金山云网络技术有限公司 | Processing method, device and system for distributed form storage |
| CN111444207B (en) * | 2020-03-24 | 2024-03-01 | 广州虎牙科技有限公司 | Method and device for changing node data, storage medium and server |
| CN111460034B (en) * | 2020-03-25 | 2024-02-23 | 聚好看科技股份有限公司 | Bidirectional database synchronization method, synchronization unit and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013078689A1 (en) * | 2011-12-02 | 2013-06-06 | 华为技术有限公司 | Method and device for realizing message transfer in cloud message service |
| CN103838642B (en) * | 2012-11-26 | 2018-03-13 | 腾讯科技（深圳）有限公司 | A kind of data reconstruction method, device and system |
| CN104376017B (en) * | 2013-08-15 | 2018-10-23 | 阿里巴巴集团控股有限公司 | The method and system of data synchronization are carried out between database |
| CN114153702A (en) * | 2015-04-03 | 2022-03-08 | 甲骨文国际公司 | Method and system for implementing a log parser in a log analysis system |
-
2017
- 2017-01-24 CN CN201710060057.4A patent/CN108345617B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
Non-Patent Citations (1)
| Title |
|---|
| Optimization of Heterogeneous Databases Data Synchronization in WAN by Virtual Log Compression;Zhenyu Lu 等;《2010 Second International Conference on Future Networks》;20100318;第98-101页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108345617A (en) | 2018-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108345617B (en) | Data synchronization method and device and electronic equipment | |
| AU2017282817B2 (en) | Data processing method and device | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| CN110990432A (en) | Device and method for synchronizing distributed cache clusters across machine rooms | |
| CN106452836B (en) | main node setting method and device | |
| CN110022333B (en) | Communication method and device of distributed system | |
| CN112000635A (en) | Data request method, device and medium | |
| CN110099084B (en) | Method, system and computer readable medium for ensuring storage service availability | |
| RU2653254C1 (en) | Method, node and system for managing data for database cluster | |
| TW201525718A (en) | Block storage gateway module, mediator system for storage, cloud storage system, method for providing access to block storage, mediating method for storage, and content delivery apparatus | |
| CN107612950A (en) | A kind of method, apparatus, system, electronic equipment that service is provided | |
| CN115277727B (en) | Data disaster recovery method, system, device and storage medium | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| JP5366858B2 (en) | Cluster system and system switching method in cluster system | |
| CN103856511A (en) | Data packet upload method, client terminal, nodes, information server and system | |
| CN113326251B (en) | Data management method, system, device and storage medium | |
| CN107528703B (en) | Method and equipment for managing node equipment in distributed system | |
| CN114625566A (en) | Data disaster tolerance method and device, electronic equipment and storage medium | |
| WO2015196692A1 (en) | Cloud computing system and processing method and apparatus for cloud computing system | |
| CN110351122B (en) | Disaster recovery method, device, system and electronic equipment | |
| CN111083160A (en) | Resource information recovery method and device | |
| CN113596195B (en) | Public IP address management method, device, main node and storage medium | |
| CN110603798A (en) | Resilient consistency high availability in multiple single boards | |
| CN112527561A (en) | Data backup method and device based on Internet of things cloud storage | |
| CN111581013A (en) | System information backup and reconstruction method based on metadata and shadow files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |