CN110737719A - Data synchronization method, device, equipment and computer readable storage medium - Google Patents

Data synchronization method, device, equipment and computer readable storage medium Download PDF

Info

Publication number
CN110737719A
CN110737719A CN201910842695.0A CN201910842695A CN110737719A CN 110737719 A CN110737719 A CN 110737719A CN 201910842695 A CN201910842695 A CN 201910842695A CN 110737719 A CN110737719 A CN 110737719A
Authority
CN
China
Prior art keywords
block
data
snapshot
attribute information
slave node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910842695.0A
Other languages
Chinese (zh)
Other versions
CN110737719B (en
Inventor
刘子锐
谷忠言
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Ping An Communication Technology Co Ltd
Original Assignee
Shenzhen Ping An Communication Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Ping An Communication Technology Co Ltd filed Critical Shenzhen Ping An Communication Technology Co Ltd
Priority to CN201910842695.0A priority Critical patent/CN110737719B/en
Publication of CN110737719A publication Critical patent/CN110737719A/en
Application granted granted Critical
Publication of CN110737719B publication Critical patent/CN110737719B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275—Synchronous replication
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The method comprises the steps of determining a snapshot block set needing to be synchronized by comparing attribute information of a current snapshot block corresponding to each th data block on a received target slave node with second attribute information of a current snapshot block corresponding to each local second data block, and simultaneously sending the snapshot block set needing to be synchronized to the target slave node, so that the target slave node can update the corresponding data block based on each snapshot block in the snapshot block set, and accordingly snapshots needing to be transmitted and updated can be flexibly controlled according to actual conditions.

Description

Data synchronization method, device, equipment and computer readable storage medium
Technical Field
The present application relates to the field of data processing, and in particular, to data synchronization method, apparatus, device, and computer-readable storage medium.
Background
In the big data era, along with the continuous increase of data size and complexity, the performance requirement of the distributed storage system is higher and higher.
In general, when the distributed storage system fails, or several nodes are in an offline state, and after the failure disappears, when the node or the nodes are in an online state again, the data in the node or the nodes generally needs to be synchronized to ensure the consistency of the data among all the nodes.
In the conventional method, a full replication method is usually adopted to synchronize data on a master node to a slave node which is in an online state again, and in the above case, because the data volume of data to be synchronized is huge, it takes a long time to complete the synchronization process of the data, thereby resulting in low efficiency of data synchronization, and because the data volume of the transmitted data is large, the burden of network bandwidth is increased.
Therefore, how to improve the efficiency of data synchronization and reduce the burden of network bandwidth is an urgent problem to be solved at present.
Disclosure of Invention
The present application mainly aims to provide data synchronization methods, apparatuses, devices and computer readable storage media, aiming to improve the efficiency of data synchronization and reduce the burden of network bandwidth.
, the present application provides methods of data synchronization, the methods comprising the steps of:
receiving attribute information of a current snapshot block corresponding to each data block sent by a target slave node;
acquiring second attribute information of a current snapshot block corresponding to each second data block;
determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
and sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
In a second aspect, the present application further provides data synchronization apparatuses, including:
a receiving module, configured to receive -th attribute information of a current snapshot block corresponding to each -th data block sent by a target slave node;
the acquisition module is used for acquiring second attribute information of the current snapshot block corresponding to each second data block;
the determining module is used for determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
and the sending module is used for sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
In a third aspect, the present application further provides computer apparatuses comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the data synchronization method as described above.
In a fourth aspect, the present application further provides computer-readable storage media having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the steps of the data synchronization method as described above.
The application provides data synchronization methods, apparatuses, devices, and computer-readable storage media, which determine a snapshot block set that needs synchronization by comparing attribute information of a current snapshot block corresponding to each -th data block on a target slave node with second attribute information of a current snapshot block corresponding to each second local data block, and send the snapshot block set that needs synchronization to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set, and can flexibly control snapshots that need to be transmitted and updated according to actual conditions.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of an data synchronization method according to an embodiment of the present application;
fig. 2 is a schematic flow chart of another data synchronization method according to an embodiment of the present application;
fig. 3 is a schematic flow chart of another data synchronization method according to an embodiment of the present application;
fig. 4 is a schematic flow chart of another data synchronization method provided in the present application;
fig. 5 is a schematic flow chart of another data synchronization method according to an embodiment of the present application;
FIG. 6 is a schematic block diagram of data synchronization devices provided in the embodiments of the present application;
fig. 7 is a schematic block diagram of another data synchronization apparatus provided in an embodiment of the present application;
fig. 8 is a schematic block diagram of another data synchronization apparatus provided in an embodiment of the present application;
fig. 9 is a schematic block diagram of another data synchronization apparatus provided in an embodiment of the present application;
fig. 10 is a schematic block diagram of another data synchronization apparatus provided in an embodiment of the present application;
fig. 11 is a block diagram illustrating a structure of a computer device according to an embodiment of the present application .
The objectives, features, and advantages of the present application will be further explained in connection with the embodiments, as illustrated in the accompanying drawings at .
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all embodiments of .
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
The embodiment of the application provides data synchronization methods, devices, computer equipment and computer readable storage media, wherein the data synchronization method can be applied to terminal equipment or a server, the terminal equipment can be electronic equipment such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant and wearable equipment, and the server can be a single server or a server cluster consisting of a plurality of servers.
The embodiments of the present application are described in detail below with reference to the drawings, and features of the following examples and embodiments may be combined without conflict.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an data synchronization method according to an embodiment of the present application.
As shown in fig. 1, the data synchronization method includes steps S101 to S104.
And S101, receiving attribute information of a current snapshot block corresponding to each th data block sent by the target slave node.
Specifically, after a target slave node in the distributed storage system is in the offline state , when the target slave node is in the online state again, in order to ensure that data on the target slave node is up-to-date, that is, in order to ensure consistency of data of the master node and the slave node, -th attribute information of a current snapshot block corresponding to each -th data block stored on the target slave node needs to be sent to a server, that is, -th attribute information of a current snapshot block corresponding to each -th data block which is cut in advance and stored on the target slave node needs to be sent to a master node, so that preparation is made for quickly determining a snapshot to be synchronized subsequently, wherein the target slave node is any nodes except the master node in the distributed storage system.
The distributed storage system is used for storing data in a distributed mode on a plurality of independent devices, independent devices are called nodes, usually, main nodes and a plurality of slave nodes, the distributed storage system can be an HDFS (Hadoop distributed file system), the HDFS is a big data distributed file system, a current snapshot block is a current snapshot block corresponding to each th data block, a th data block is a data block on a pre-segmented target slave node, and th attribute information is attribute information of the current snapshot block.
It should be noted that the current snapshot Block represents state records of moments before each th data Block, that is, all current snapshot blocks represent history state records of each 0 th data Block, the 1 th attribute information includes a 2 th version number and a 3 th identifier, and may further include other information, the 4 th version number is the version number of the current snapshot Block of the th data Block, for example, the th version number may be "V3", the th identifier is an identifier of the current snapshot Block of the th data Block, the th identifier is used to uniquely identify the current snapshot Block of the th data Block, for example, the th identifier may be "Block 1".
It is understood that the th version number and the th identifier can be pure letters, pure numbers, or combinations of letters and/or numbers, etc., and the specific contents of the distributed storage system, the th version number and the th identifier can be set according to practical applications, and are not limited herein.
And step S102, acquiring second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to analyze whether the snapshot blocks on the target slave node and the master node are consistent, the server needs to obtain each storage path of second attribute information of a current snapshot block corresponding to each local second data block from the attribute database, and then extract the second attribute information according to each storage path of the second attribute information, where the second data block is a data block pre-split on the master node, the second attribute information is attribute information of a current snapshot block corresponding to each second data block, the second attribute information includes a second version number and a second identifier, and may further include other information, the second version number is a version number of the current snapshot block of the second data block, the second identifier is an identifier of the current snapshot block of the second data block, and the second identifier is used for identifying the current snapshot block of the second data block only .
It should be noted that the attribute database may be an SQL database or an oracle database, and the specific contents of the attribute database, the second version number, and the second identifier may be set according to practical applications, which is not limited herein.
Step S103, determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to synchronize snapshot blocks needing to be synchronized into a target node, a snapshot block set to be synchronized needs to be determined according to attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block, and specifically includes that according to each th identifier and each second identifier, a current snapshot block corresponding to a second data block with the same second identifier as the th identifier is collected to form a candidate snapshot block set, according to a second version number of each snapshot block in the candidate snapshot block set and a th version number of the current snapshot block corresponding to each th data block, a th snapshot set to be synchronized is determined, snapshot blocks except the current snapshot block corresponding to the second data block in the candidate snapshot block set are used as the second snapshot set to be synchronized locally, and a union of the snapshot set and the second snapshot set is used as the snapshot set to be synchronized.
The snapshot blocks with the same identifier are used as a candidate snapshot block set, then the snapshot blocks with different version numbers in the candidate snapshot block set are used as th snapshot sets to be synchronized, next, the snapshot blocks except the candidate snapshot block set are used as second snapshot sets to be synchronized, and finally, the th snapshot set and the second snapshot set are used as total snapshot sets to be synchronized, so that the efficiency of determining the snapshot sets to be synchronized is improved.
For example, the -th attribute information includes "Block 1V 3", "Block 2V 1" and "Block 4V 3", the second attribute information includes "Block 1V 3", "Block 2V 3", "Block 3V 3" and "Block 4V 3", the second identification and the -th identification are the same as the "Block 1V 3", "Block 2V 3" and "Block 4V 3" as candidate snapshot Block sets, the -th version and the second version of "Block 2V 3" are determined as the -th snapshot set to be synchronized, the "Block 3V 3" is determined as the second snapshot set to be synchronized, and finally the "Block 2V 3" and the "Block 3V 3" are determined as the snapshot sets to be synchronized.
And step S104, sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Specifically, in order to ensure the consistency of the data between the master node and the slave node, the snapshot block set determined in step S102 needs to be sent to the target slave node, and when the target slave node receives the snapshot block set, the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set, that is, the target slave node overwrites the original snapshot block with each snapshot block in the received snapshot block set to obtain the latest snapshot block, and then updates the latest snapshot block to the corresponding data block, thereby obtaining the latest data.
And , updating the latest snapshot block to the corresponding data block to obtain the latest data, specifically including sending the latest log to the target slave node by the server after sending the snapshot block set to the target slave node, and then updating the latest snapshot block to the corresponding data block according to the chronological order of the log items in the latest log to obtain the latest data.
In the data synchronization method provided in the above embodiment, the attribute information of the current snapshot block corresponding to each th data block on the target slave node is compared with the acquired second attribute information of the current snapshot block corresponding to each second local data block, so as to determine the snapshot block set that needs to be synchronized, and simultaneously, the snapshot block set that needs to be synchronized is sent to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set, and thus, the snapshot that needs to be transmitted and updated can be flexibly controlled according to the actual situation, that is, the snapshot that needs to be synchronized is transmitted and updated, and the synchronized snapshot is not required to be transmitted and updated, so that the synchronization time is reduced, and the efficiency of data synchronization is improved.
Referring to fig. 2, fig. 2 is a flowchart illustrating another data synchronization method according to an embodiment of the present application.
As shown in fig. 2, the data synchronization method includes steps S201 to 206.
Step S201, attribute information of the current snapshot block corresponding to each th data block sent by the target slave node is received.
Specifically, after a target slave node in the distributed storage system is in the offline state , when the target slave node is in the online state again, in order to ensure that data on the target slave node is up-to-date, that is, in order to ensure consistency of data of the master node and the slave node, -th attribute information of a current snapshot block corresponding to each -th data block stored on the target slave node needs to be sent to a server, that is, -th attribute information of a current snapshot block corresponding to each -th data block which is cut in advance and stored on the target slave node needs to be sent to a master node, so that preparation is made for quickly determining a snapshot to be synchronized subsequently, wherein the target slave node is any nodes except the master node in the distributed storage system.
Step S202, second attribute information of the current snapshot block corresponding to each second data block is obtained.
Specifically, in order to analyze whether the snapshot blocks on the target slave node and the master node are consistent or not, the server needs to obtain each storage path of second attribute information of a current snapshot block corresponding to each second local data block from the attribute database, and then extract the second attribute information according to each storage path, where the second data block is a data block pre-split on the master node, the second attribute information is attribute information of a current snapshot block corresponding to each second data block, the second attribute information includes a second version number and a second identifier, and may further include other information, the second version number is a version number of the current snapshot block of the second data block, the second identifier is an identifier of the current snapshot block of the second data block, and the second identifier is used to uniquely identify the current snapshot block of the second data block .
Step S203, determining whether to trigger a data synchronization instruction according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to determine whether synchronization of snapshot blocks to a target slave node is required, it is required to determine whether a data synchronization instruction is triggered according to th attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block, specifically, the method includes obtaining th version number and th identifier of a current snapshot block corresponding to each th data block from th attribute information of a current snapshot block corresponding to each th data block, obtaining a second version number and a second identifier of a current snapshot block corresponding to each second data block from second attribute information of a current snapshot block corresponding to each second data block, determining whether each th identifier is the same as each second identifier, determining whether each th version number is the same as each second version number, determining whether each th identifier is the same as each second snapshot block and each second version number, determining that no data synchronization instruction exists if each th identifier is the same as each second snapshot block and each second identifier is the same as each second identifier, determining that no data synchronization instruction exists if each th identifier is the second snapshot block is the same as each second snapshot block and each second identifier is the second snapshot block is the same as each second identifier, determining that no 3638 and at least the second synchronization instruction exists, and at least triggering no 3636 is different from 3638, and 3638.
Step S204, if a data synchronization instruction is triggered, determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to synchronize snapshot blocks needing to be synchronized into a target node, a snapshot block set to be synchronized needs to be determined according to attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block, and specifically includes that according to each th identifier and each second identifier, a current snapshot block corresponding to a second data block with the same second identifier as the th identifier is collected to form a candidate snapshot block set, according to a second version number of each snapshot block in the candidate snapshot block set and a th version number of the current snapshot block corresponding to each th data block, a th snapshot set to be synchronized is determined, snapshot blocks except the current snapshot block corresponding to the second data block in the candidate snapshot block set are used as the second snapshot set to be synchronized locally, and a union of the snapshot set and the second snapshot set is used as the snapshot set to be synchronized.
Step S205, sending the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Specifically, in order to ensure the consistency of the data between the master node and the slave node, the snapshot block set determined in step S204 needs to be sent to the target slave node, and when the target slave node receives the snapshot block set, the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set, that is, the target slave node overwrites the original snapshot block with each snapshot block in the received snapshot block set to obtain the latest snapshot block, and then updates the latest snapshot block to the corresponding data block, thereby obtaining the latest data.
Step S206, if the data synchronization instruction is not triggered, sending corresponding response information to the target slave node, so that the target slave node updates data based on the current snapshot block corresponding to each -th data block.
Specifically, the master node stores logs in real time, the logs are used for recording all records of data block updating, because the target node is switched from an offline state to an online state, and a latest log is not stored, when a data synchronization instruction is not triggered, the server needs to send corresponding response information to the target slave node, that is, the server needs to send a corresponding latest log to the target slave node, and when the target slave node receives the latest log, each data block is updated according to the latest log and a current snapshot block corresponding to each data block, so as to obtain the latest data.
According to the data synchronization method provided by the embodiment, when the snapshot block needs to be synchronized, the synchronization instruction is triggered, the snapshot block is synchronized to the target slave node, when the snapshot block does not need to be synchronized, the synchronization instruction is not triggered, the latest log is synchronized to the target slave node, so that the target slave node updates data according to the received snapshot block and/or the latest log, the data needing to be synchronized is sent to the target slave node according to the actual situation, the situation that the whole amount of data is sent to the target slave node in the traditional method is avoided, and the data updating efficiency is improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating another data synchronization method according to an embodiment of the present application.
As shown in fig. 3, the data synchronization method includes steps S301 to 307.
Step S301, receiving an online request initiated by a target slave node, wherein the online request includes node online information corresponding to the target slave node.
Specifically, in the distributed storage system, due to a failure or the like, a target slave node, that is, a slave node, may be in an offline state or an online state, when the target slave node is in the offline state and wants to be in the online state again, online requests need to be initiated to the server, and the server receives the online requests initiated by the target slave node in real time or within a preset time period, where the online requests are requests sent by the target slave node to the server for applying for being in the online state.
It should be noted that the node account number may be a pure letter, or a letter and/or a number combination, etc., the password may be a pure number, or a letter and/or a number combination, etc., the node identifier identifies the target slave node, the node identifier may be a pure number, or a letter and/or a number combination, etc., and the verification code may be a graphic verification, a numerical verification, a text verification, etc.
Step S302, checking the node online information to obtain a checking result whether the checking is passed or not.
When a server, that is, a master node, receives an online request initiated by a target slave node, in order to ensure the security of the distributed storage system, authentication of the target slave node needs to be performed, that is, the server first obtains target online information in a preset online database, then the server determines whether the obtained target online information and the received node online information are -fold, if -fold, it is determined that the verification result is a verification pass, that is, it is determined that the target slave node passes the authentication, and if -fold, it is determined that the verification result is a verification fail, that is, it is determined that the target slave node fails the authentication, and meanwhile, prompt information indicating that the verification fails is output, so that the security of the distributed storage system is improved.
And step S303, when the verification result is that the verification is passed, sending th attribute information acquisition request to the target slave node.
Specifically, when the verification result is that the verification is passed, that is, when it is determined that the target slave node passes the authentication of the target slave node, the server sends, to the target slave node, an acquisition request of the th attribute information of the current snapshot block corresponding to each th data block, where the contents of the th data block, the current snapshot block, and the th attribute information in step S303 correspond to the contents of the th data block, the current snapshot block, and the th attribute information in step S301, which is not described herein.
Step S304, receiving the attribute information of the current snapshot block corresponding to each th data block sent by the target slave node.
Specifically, after a target slave node in the distributed storage system is in the offline state , when the target slave node is in the online state again, in order to ensure that data on the target slave node is up-to-date, that is, in order to ensure consistency of data of the master node and the slave node, -th attribute information of a current snapshot block corresponding to each -th data block stored on the target slave node needs to be sent to a server, that is, -th attribute information of a current snapshot block corresponding to each -th data block which is cut in advance and stored on the target slave node needs to be sent to a master node, so that preparation is made for quickly determining a snapshot to be synchronized subsequently, wherein the target slave node is any nodes except the master node in the distributed storage system.
Step S305, obtain second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to analyze whether the snapshot blocks on the target slave node and the master node are consistent or not, the server needs to obtain each storage path of second attribute information of a current snapshot block corresponding to each second local data block from the attribute database, and then extract the second attribute information according to each storage path, where the second data block is a data block pre-split on the master node, the second attribute information is attribute information of a current snapshot block corresponding to each second data block, the second attribute information includes a second version number and a second identifier, and may further include other information, the second version number is a version number of the current snapshot block of the second data block, the second identifier is an identifier of the current snapshot block of the second data block, and the second identifier is used to uniquely identify the current snapshot block of the second data block .
Step S306, determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, in order to determine whether synchronization of snapshot blocks to a target slave node is required, it is required to determine whether a data synchronization instruction is triggered according to th attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block, specifically, the method includes obtaining th version number and th identifier of a current snapshot block corresponding to each th data block from th attribute information of a current snapshot block corresponding to each th data block, obtaining a second version number and a second identifier of a current snapshot block corresponding to each second data block from second attribute information of a current snapshot block corresponding to each second data block, determining whether each th identifier is the same as each second identifier, determining whether each th version number is the same as each second version number, determining whether each th identifier is the same as each second snapshot block and each second version number, determining that no data synchronization instruction exists if each th identifier is the same as each second snapshot block and each second identifier is the same as each second identifier, determining that no data synchronization instruction exists if each th identifier is the second snapshot block is the same as each second snapshot block and each second identifier is the second snapshot block is the same as each second identifier, determining that no 3638 and at least the second synchronization instruction exists, and at least triggering no 3636 is different from 3638, and 3638.
Step S307, the snapshot block set is sent to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Specifically, in order to ensure the consistency of the data between the master node and the slave node, the snapshot block set determined in step S204 needs to be sent to the target slave node, and when the target slave node receives the snapshot block set, the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set, that is, the target slave node overwrites the original snapshot block with each snapshot block in the received snapshot block set to obtain the latest snapshot block, and then updates the latest snapshot block to the corresponding data block, thereby obtaining the latest data.
According to the data synchronization method provided by the embodiment, the target slave node needing to be online is verified, and the th attribute information is received after the verification is passed, so that the safety of acquiring the th attribute information is improved, and the safety of data synchronization can be effectively improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating another data synchronization method according to an embodiment of the present application.
As shown in fig. 4, the data synchronization method includes steps S401 to 408.
Step S401, attribute information of the current snapshot block corresponding to each th data block sent by the target slave node is received.
Specifically, in the distributed storage system, due to a failure or the like, a target slave node, that is, a slave node, may be in an offline state or an online state, when the target slave node is in the offline state and wants to be in the online state again, online requests need to be initiated to the server, and the server receives the online requests initiated by the target slave node in real time or within a preset time period, where the online requests are requests sent by the target slave node to the server for applying for being in the online state.
It should be noted that the node account number may be a pure letter, or a letter and/or a number combination, etc., the password may be a pure number, or a letter and/or a number combination, etc., the node identifier identifies the target slave node, the node identifier may be a pure number, or a letter and/or a number combination, etc., and the verification code may be a graphic verification, a numerical verification, a text verification, etc.
Step S402, second attribute information of the current snapshot block corresponding to each second data block is obtained.
When a server, that is, a master node, receives an online request initiated by a target slave node, in order to ensure the security of the distributed storage system, authentication of the target slave node needs to be performed, that is, the server first obtains target online information in a preset online database, then the server determines whether the obtained target online information and the received node online information are -fold, if -fold, it is determined that the verification result is a verification pass, that is, it is determined that the target slave node passes the authentication, and if -fold, it is determined that the verification result is a verification fail, that is, it is determined that the target slave node fails the authentication, and meanwhile, prompt information indicating that the verification fails is output, so that the security of the distributed storage system is improved.
And S403, determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, when the verification result is that the verification is passed, that is, when it is determined that the target slave node passes the authentication, the server sends, to the target slave node, an acquisition request of the th attribute information of the current snapshot block corresponding to each th data block, where the contents of the th data block, the current snapshot block, and the th attribute information in step S403 correspond to the contents of the th data block, the current snapshot block, and the th attribute information in step S401, which is not described herein.
Step S404, the snapshot block set is sent to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Specifically, after a target slave node in the distributed storage system is in the offline state , when the target slave node is in the online state again, in order to ensure that data on the target slave node is up-to-date, that is, in order to ensure consistency of data of the master node and the slave node, -th attribute information of a current snapshot block corresponding to each -th data block stored on the target slave node needs to be sent to a server, that is, -th attribute information of a current snapshot block corresponding to each -th data block which is cut in advance and stored on the target slave node needs to be sent to a master node, so that preparation is made for quickly determining a snapshot to be synchronized subsequently, wherein the target slave node is any nodes except the master node in the distributed storage system.
Step S405, obtaining a current system time and an end time, where the end time is a time when recording of the current snapshot block corresponding to each second data block is finished.
Specifically, the server records each time of performing the action for analyzing the data, and in order to analyze the latest log, it needs to obtain a current system time and an end time, so as to facilitate subsequent analysis, where the current system time is a current time of the system, for example, the current system time may be "10 minutes 10 seconds at 1 month 1 day 10 of 2019", and the end time is a time when recording of the current snapshot block corresponding to each second data block is finished.
And step S406, determining a log recording time range according to the current system time and the end time.
Specifically, after the current system time and the end time are acquired, the difference between the current system time and the end time is determined as a logging time range, or the absolute value of the difference between the current system time and the end time is determined as the logging time range. For example, if the current system time is "10 minutes 10 seconds at 1 month and 1 day in 2019" and the end time is "8 minutes 5 seconds at 10 points 1 month and 1 day in 2019" and the end time is "2 minutes", the "2 minutes" is determined as the logging time range.
Step S407 determines the log whose recording time is within the log recording time range as the latest log.
Specifically, after the log recording time range is determined, a target log having a recording time within the log recording time range determined in step S406 is acquired from the log database, and the target log is determined to be the newest log.
Step S408, the latest log is sent to the target slave node, so that the target slave node synchronizes the data blocks based on the latest log and the snapshot block set.
Specifically, the latest log is sent to the target slave node, and when the target slave node receives the latest log, the latest snapshot block is updated to the corresponding data block according to the time sequence of the log entry in the latest log and the snapshot block set, so that the latest data is obtained.
According to the data synchronization method provided by the embodiment, the latest log and the snapshot block needing to be updated are sent to the target slave node, and when the target slave node receives the latest log and the snapshot block needing to be updated, data is updated, so that consistency of data between the master node and the slave node is guaranteed.
Referring to fig. 5, fig. 5 is a flowchart illustrating another data synchronization method according to an embodiment of the present application.
As shown in fig. 5, the data synchronization method includes steps S501 to 507.
Step S501, target data is obtained, and target capacity of the target data is calculated.
Specifically, in order to perform segmentation processing on target data, the target data needs to be obtained in a data engine, and then a target capacity of the target data is calculated, where the data engine is an engine in which a target slave node provides processing transactions for a client and meets an access requirement of the client, and the target capacity is a capacity of the target data.
And step S502, determining the segmentation capacity according to the target capacity and the preset segmentation quantity.
Specifically, a preset segmentation number is obtained in a segmentation database, a quotient and a remainder between the target capacity and the segmentation number are calculated, if the remainder is a number zero, the quotient is used as the segmentation capacity, and if the remainder is not the number zero, the sum of the quotient and the number is used as the segmentation capacity.
And S503, carrying out segmentation processing on the target data according to the segmentation capacity to obtain a plurality of data blocks.
Specifically, according to the determined splitting capacity in step S502, the target data obtained in step S501 is split from the start end or the end of the target data, so as to obtain a plurality of data blocks.
For example, a start end of the target data is selected, then a data block with a segmentation capacity is extracted from the target data from the start end of the target data to obtain the extracted target data, then the start end of the extracted target data is selected, and then the data block with the segmentation capacity is extracted from the extracted target data from the start end of the extracted target data until the target data is segmented into a plurality of data blocks.
Step S504, attribute information of the current snapshot block corresponding to each th data block sent by the target slave node is received.
Specifically, in the distributed storage system, due to a failure or the like, a target slave node, that is, a slave node, may be in an offline state or an online state, when the target slave node is in the offline state and wants to be in the online state again, online requests need to be initiated to the server, and the server receives the online requests initiated by the target slave node in real time or within a preset time period, where the online requests are requests sent by the target slave node to the server for applying for being in the online state.
It should be noted that the node account number may be a pure letter, or a letter and/or a number combination, etc., the password may be a pure number, or a letter and/or a number combination, etc., the node identifier identifies the target slave node, the node identifier may be a pure number, or a letter and/or a number combination, etc., and the verification code may be a graphic verification, a numerical verification, a text verification, etc.
And step S505, obtaining second attribute information of the current snapshot block corresponding to each second data block.
When a server, that is, a master node, receives an online request initiated by a target slave node, in order to ensure the security of the distributed storage system, authentication of the target slave node needs to be performed, that is, the server first obtains target online information in a preset online database, then the server determines whether the obtained target online information and the received node online information are -fold, if -fold, it is determined that the verification result is a verification pass, that is, it is determined that the target slave node passes the authentication, and if -fold, it is determined that the verification result is a verification fail, that is, it is determined that the target slave node fails the authentication, and meanwhile, prompt information indicating that the verification fails is output, so that the security of the distributed storage system is improved.
Step S506, determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block.
Specifically, when the verification result is that the verification is passed, that is, when it is determined that the target slave node has passed the authentication, the server sends, to the target slave node, an acquisition request of the th attribute information of the current snapshot block corresponding to each th data block, where the contents of the th data block, the current snapshot block, and the th attribute information in step S506 are identical to the contents of the th data block, the current snapshot block, and the th attribute information in step S504, which is not described herein.
Step S507, sending the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Specifically, after a target slave node in the distributed storage system is in the offline state , when the target slave node is in the online state again, in order to ensure that data on the target slave node is up-to-date, that is, in order to ensure consistency of data of the master node and the slave node, -th attribute information of a current snapshot block corresponding to each -th data block stored on the target slave node needs to be sent to a server, that is, -th attribute information of a current snapshot block corresponding to each -th data block which is cut in advance and stored on the target slave node needs to be sent to a master node, so that preparation is made for quickly determining a snapshot to be synchronized subsequently, wherein the target slave node is any nodes except the master node in the distributed storage system.
According to the data synchronization method provided by the embodiment, the target data is divided into the plurality of data blocks, and the snapshot block to be synchronized is determined according to the attribute information corresponding to each data block, so that at most all the snapshot blocks can be synchronized to the target slave node according to actual conditions, the synchronization efficiency of the snapshot blocks is improved, the data synchronization efficiency is improved, and the situation that the snapshot blocks are fully synchronized in the traditional method is avoided.
Referring to fig. 6, fig. 6 is a schematic block diagram of data synchronization apparatuses according to an embodiment of the present application.
As shown in fig. 6, the data synchronization apparatus 600 includes: a receiving module 601, an obtaining module 602, a determining module 603 and a sending module 604.
A receiving module 601, configured to receive -th attribute information of a current snapshot block corresponding to each -th data block sent by a target slave node;
an obtaining module 602, configured to obtain second attribute information of a current snapshot block corresponding to each second data block;
the determining module 603 is configured to determine a snapshot block set to be synchronized according to the -th attribute information of the current snapshot block corresponding to each -th data block and the second attribute information of the current snapshot block corresponding to each second data block;
the sending module 604 is configured to send the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
In , the determining module 603 is further configured to collect, according to each identifier and each second identifier, a current snapshot block corresponding to a second data block with a second identifier that is the same as the identifier to form a candidate snapshot block set, determine a th snapshot set to be synchronized according to a second version number of each snapshot block in the candidate snapshot block set and a th version number of the current snapshot block corresponding to each th data block, use snapshot blocks in the local other than the current snapshot block corresponding to the second data block in the candidate snapshot block set as the second snapshot set to be synchronized, and use a union of the th snapshot set and the second snapshot set as the snapshot set to be synchronized.
Referring to fig. 7, fig. 7 is a schematic block diagram of another data synchronization apparatus according to an embodiment of the present application.
As shown in fig. 7, the data synchronization apparatus 700 includes a receiving module 701, an obtaining module 702, an th determining module 703, a triggering module 704, a second determining module 705, and a sending module 706.
A receiving module 701, configured to receive -th attribute information of a current snapshot block corresponding to each -th data block sent by a target slave node;
an obtaining module 702, configured to obtain second attribute information of a current snapshot block corresponding to each second data block;
an determining module 703, configured to determine whether to trigger a data synchronization instruction according to the th attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
a triggering module 704, configured to trigger a second determining module 705 if the data synchronization instruction is triggered;
the second determining module 705 is configured to determine a snapshot block set to be synchronized according to the -th attribute information of the current snapshot block corresponding to each -th data block and the second attribute information of the current snapshot block corresponding to each second data block;
a sending module 706, configured to send the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set;
the sending module 706 is further configured to send corresponding response information to the target slave node if the data synchronization instruction is not triggered, so that the target slave node updates the data based on the current snapshot block corresponding to each -th data block.
In , the determining module 703 is further configured to obtain 3 th version number and 4 th identifier of each 2 th data block from 1 th attribute information of a current snapshot block corresponding to each 0 th data block, obtain second and second identifiers of each th data block corresponding to a current snapshot block from second attribute information of a current snapshot block corresponding to each second data block, determine whether each 5 th identifier is the same as each second identifier, and determine whether each 6 th version number is the same as each second version number, determine not to trigger the data synchronization instruction if each th identifier is the same as each second identifier and each th version number is the same as each second version number, determine to trigger the data synchronization instruction if at least th identifiers are different from at least second identifiers and/or at least th version numbers are different from at least second version numbers.
Referring to fig. 8, fig. 8 is a schematic block diagram of another data synchronization apparatus according to an embodiment of the present application.
As shown in fig. 8, the data synchronization apparatus 800 includes: a receiving module 801, a checking module 802, a sending module 803, an obtaining module 804 and a determining module 805.
A receiving module 801, configured to receive an online request initiated by a target slave node, where the online request includes node online information corresponding to the target slave node;
the checking module 802 is configured to check the node connection information to obtain a check result indicating whether the check is passed;
a sending module 803, configured to send an th attribute information obtaining request to the target slave node when the verification result is that the verification passes;
the receiving module 801 is further configured to receive attribute information of a current snapshot block corresponding to each th data block sent by the target slave node;
an obtaining module 804, configured to obtain second attribute information of a current snapshot block corresponding to each second data block;
a determining module 805, configured to determine, according to the -th attribute information of the current snapshot block corresponding to each -th data block and the second attribute information of the current snapshot block corresponding to each second data block, a snapshot block set to be synchronized;
the sending module 803 is further configured to send the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Referring to fig. 9, fig. 9 is a schematic block diagram of another data synchronization apparatus according to an embodiment of the present application.
As shown in fig. 9, the data synchronization apparatus 900 includes a receiving module 901, an obtaining module 902, a determining module 903, and a sending module 904.
A receiving module 901, configured to receive attribute information of a current snapshot block corresponding to each th data block sent by a target slave node;
an obtaining module 902, configured to obtain second attribute information of a current snapshot block corresponding to each second data block;
a determining module 903, configured to determine a snapshot block set to be synchronized according to the -th attribute information of the current snapshot block corresponding to each -th data block and the second attribute information of the current snapshot block corresponding to each second data block;
a sending module 904, configured to send the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set;
the obtaining module 902 is further configured to obtain a current system time and an end time, where the end time is a time when recording of the current snapshot block corresponding to each second data block is finished;
the determining module 903 is further configured to determine a log recording time range according to the current system time and the end time;
the determining module 903 is further configured to determine a log with a recording time within a log recording time range as a latest log;
the sending module 904 is further configured to send the latest log to the target slave node, so that the target slave node synchronizes the data block based on the latest log and the snapshot block set.
Referring to fig. 10, fig. 10 is a schematic block diagram of another data synchronization apparatus according to an embodiment of the present application.
As shown in fig. 10, the data synchronization apparatus 1000 includes: an obtaining module 1001, a determining module 1002, a cutting module 1003, a receiving module 1004, and a transmitting module 1005.
An obtaining module 1001, configured to obtain target data and calculate a target capacity of the target data;
a determining module 1002, configured to determine a segmentation capacity according to the target capacity and a preset segmentation number;
a segmentation module 1003, configured to perform segmentation processing on the target data according to the segmentation capacity to obtain a plurality of data blocks;
a receiving module 1004, configured to receive attribute information of a current snapshot block corresponding to each th data block sent by the target slave node;
an obtaining module 1001, configured to obtain second attribute information of a current snapshot block corresponding to each second data block;
a determining module 1002, configured to determine a snapshot block set to be synchronized according to the -th attribute information of the current snapshot block corresponding to each -th data block and the second attribute information of the current snapshot block corresponding to each second data block;
a sending module 1005, configured to send the snapshot block set to the target slave node, so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
It should be noted that, as will be clear to those skilled in the art, for convenience and brevity of description, the specific working processes of the apparatus and each module and unit described above may refer to the corresponding processes in the foregoing data synchronization method embodiment, and are not described herein again.
The apparatus provided by the above embodiments may be implemented in the form of computer programs that can be run on a computer device as shown in fig. 11.
Referring to fig. 11, fig. 11 is a schematic block diagram illustrating a structure of computer devices according to an embodiment of the present application.
As shown in fig. 11, the computer device includes a processor, a memory, and a network interface connected by a system bus, wherein the memory may include a nonvolatile storage medium and an internal memory.
The computer program includes program instructions that, when executed, cause the processor to perform any of the data synchronization methods.
The processor is used for providing calculation and control capability and supporting the operation of the whole computer equipment.
The internal memory provides an environment for the execution of a computer program on a non-volatile storage medium, which when executed by the processor, causes the processor to perform any of the data synchronization methods.
The network interface is used for network communication, such as sending assigned tasks and the like. Those skilled in the art will appreciate that the architecture shown in fig. 11 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable array (FPGA) or other Programmable logic device, discrete or transistor logic, discrete hardware components, etc.
Wherein, in embodiments, the processor is configured to run the computer program stored in the memory to implement the steps of:
receiving attribute information of a current snapshot block corresponding to each data block sent by a target slave node;
acquiring second attribute information of a current snapshot block corresponding to each second data block;
determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
and sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
In embodiments, when determining the set of snapshot blocks to be synchronized according to the attribute information of the current snapshot block corresponding to each data block and the second attribute information of the current snapshot block corresponding to each second data block, the processor is configured to:
assembling current snapshot blocks corresponding to second data blocks with the second identifications identical to the identification according to each identification and each second identification to form a candidate snapshot block set;
determining a snapshot set to be synchronized according to the second version number of each snapshot block in the candidate snapshot block set and the version number of the current snapshot block corresponding to each data block;
taking the snapshot blocks except the current snapshot block corresponding to the second data block in the candidate snapshot block set in the local as a second snapshot set to be synchronized;
the union of the th snapshot set and the second snapshot set is taken as the snapshot set to be synchronized.
Wherein, in another embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps before determining the set of snapshot blocks to be synchronized according to the attribute information of the current snapshot block corresponding to each data block and the second attribute information of the current snapshot block corresponding to each second data block:
determining whether to trigger a data synchronization instruction according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
determining a snapshot block set to be synchronized according to attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block;
and if the data synchronization instruction is not triggered, sending corresponding response information to the target slave node, so that the target slave node updates the data based on the current snapshot block corresponding to each th data block.
In embodiments, the processor, when determining whether to trigger the data synchronization instruction according to the th attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block, is configured to:
acquiring version numbers and identifiers of current snapshot blocks corresponding to data blocks from attribute information of the current snapshot blocks corresponding to data blocks;
acquiring a second version number and a second identifier of the current snapshot block corresponding to each second data block from second attribute information of the current snapshot block corresponding to each second data block;
judging whether each th mark is the same as each second mark or not, and judging whether each th version number is the same as each second version number or not;
if each th ID is the same as each second ID and each th version number is the same as each second version number, determining not to trigger the data synchronization instruction;
if there are at least th identifiers that differ from at least second identifiers, and/or at least th version numbers that differ from at least second version numbers, then a data synchronization instruction is determined to be triggered.
Wherein, in another embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps before determining the set of snapshot blocks to be synchronized according to the attribute information of the current snapshot block corresponding to each data block and the second attribute information of the current snapshot block corresponding to each second data block:
receiving an online request initiated by a target slave node, wherein the online request comprises node online information corresponding to the target slave node;
checking the node online information to obtain a checking result whether the checking is passed or not;
and when the verification result is verification passing, sending th attribute information acquisition request to the target slave node.
Wherein, in another embodiment, the processor is configured to run the computer program stored in the memory to realize that after the snapshot block set is sent to the target slave node, the processor is further configured to realize the following steps:
acquiring current system time and end time, wherein the end time is the time when recording of the current snapshot block corresponding to each second data block is finished;
determining a log recording time range according to the current system time and the end time;
determining the logs with the recording time within the log recording time range as the latest logs;
and sending the latest log to the target slave node for the target slave node to synchronize the data block based on the latest log and the snapshot block set.
Wherein, in another embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps before receiving the attribute information of the current snapshot block corresponding to each th data block sent by the target slave node:
acquiring target data and calculating the target capacity of the target data;
determining the segmentation capacity according to the target capacity and the preset segmentation quantity;
and according to the segmentation capacity, performing segmentation processing on the target data to obtain a plurality of data blocks.
The present application also provides computer-readable storage media, where a computer program is stored on the computer-readable storage media, the computer program includes program instructions, and when the program instructions are executed, the method implemented by the program instructions may refer to the embodiments of the data synchronization method of the present application.
The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises the series of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments. While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1, A data synchronization method, comprising:
receiving attribute information of a current snapshot block corresponding to each data block sent by a target slave node;
acquiring second attribute information of a current snapshot block corresponding to each second data block;
determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
and sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
2. The data synchronization method as claimed in claim 1, wherein before determining the set of snapshot blocks to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block, further comprising:
determining whether to trigger a data synchronization instruction according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
determining a snapshot block set to be synchronized according to attribute information of a current snapshot block corresponding to each th data block and second attribute information of a current snapshot block corresponding to each second data block;
and if the data synchronization instruction is not triggered, sending corresponding response information to the target slave node, so that the target slave node updates data based on the current snapshot block corresponding to each th data block.
3. The data synchronization method as claimed in claim 2, wherein the determining whether to trigger the data synchronization instruction according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block comprises:
acquiring version numbers and identifiers of current snapshot blocks corresponding to data blocks from attribute information of the current snapshot blocks corresponding to data blocks;
acquiring a second version number and a second identifier of the current snapshot block corresponding to each second data block from second attribute information of the current snapshot block corresponding to each second data block;
determining whether each of the th identifications is the same as each of the second identifications, and determining whether each of the th version numbers is the same as each of the second version numbers;
determining not to trigger a data synchronization instruction if each of the th identifiers is the same as each of the second identifiers and each of the th version numbers is the same as each of the second version numbers;
determining to trigger a data synchronization instruction if there are at least of said identification differing from at least of said second identification, and/or at least of said version number differing from at least of said second version number.
4. The data synchronization method of claim 3, wherein the determining the set of snapshot blocks to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block comprises:
according to each th identification and each second identification, aggregating the current snapshot blocks corresponding to the second data blocks with the second identifications being the same as the th identification to form a candidate snapshot block set;
determining a snapshot set to be synchronized according to the second version number of each snapshot block in the candidate snapshot block set and the version number of the current snapshot block corresponding to each data block;
taking the snapshot blocks except the current snapshot block corresponding to the second data block in the candidate snapshot block set in the local as a second snapshot set to be synchronized;
and taking the union of the th snapshot set and the second snapshot set as the snapshot set to be synchronized.
5. The data synchronization method of claim 1, wherein before receiving the attribute information of the current snapshot block corresponding to each th data block sent by the target slave node, the method further comprises:
receiving an online request initiated by the target slave node, wherein the online request comprises node online information corresponding to the target slave node;
checking the node online information to obtain a checking result whether the checking is passed or not;
and when the verification result is verification passing, sending an acquisition request of the th attribute information to the target slave node.
6. The data synchronization method of claim 1, wherein after sending the set of snapshot blocks to the target slave node, further comprising:
acquiring current system time and end time, wherein the end time is the time when recording of the current snapshot block corresponding to each second data block is finished;
determining a log recording time range according to the current system time and the end time;
determining the logs with the recording time within the log recording time range as the latest logs;
and sending the latest log to the target slave node so that the target slave node synchronizes the data block based on the latest log and the snapshot block set.
7. The data synchronization method of any of claims 1-6, wherein the receiving the attribute information of the current snapshot block corresponding to each th data block sent by the target slave node is preceded by:
acquiring target data and calculating the target capacity of the target data;
determining the segmentation capacity according to the target capacity and the preset segmentation quantity;
and according to the segmentation capacity, carrying out segmentation processing on the target data to obtain a plurality of data blocks.
The data synchronizer of 8, , wherein said data synchronizer comprises:
a receiving module, configured to receive -th attribute information of a current snapshot block corresponding to each -th data block sent by a target slave node;
the acquisition module is used for acquiring second attribute information of the current snapshot block corresponding to each second data block;
the determining module is used for determining a snapshot block set to be synchronized according to the attribute information of the current snapshot block corresponding to each th data block and the second attribute information of the current snapshot block corresponding to each second data block;
and the sending module is used for sending the snapshot block set to the target slave node so that the target slave node updates the corresponding data block based on each snapshot block in the snapshot block set.
Computer device of claim 9, , characterized in that the computer device comprises a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, carries out the steps of the data synchronization method of any of claims 1 to 7, of the .
computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, wherein the computer program, when being executed by a processor, carries out the steps of the data synchronization method according to any of claims 1 to 7.
CN201910842695.0A 2019-09-06 2019-09-06 Data synchronization method, device, equipment and computer readable storage medium Active CN110737719B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910842695.0A CN110737719B (en) 2019-09-06 2019-09-06 Data synchronization method, device, equipment and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910842695.0A CN110737719B (en) 2019-09-06 2019-09-06 Data synchronization method, device, equipment and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN110737719A true CN110737719A (en) 2020-01-31
CN110737719B CN110737719B (en) 2023-11-28

Family

ID=69267551

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910842695.0A Active CN110737719B (en) 2019-09-06 2019-09-06 Data synchronization method, device, equipment and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN110737719B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111352994A (en) * 2020-02-04 2020-06-30 浙江大华技术股份有限公司 Data synchronization method and related equipment and device
CN111569413A (en) * 2020-05-14 2020-08-25 北京代码乾坤科技有限公司 Data synchronization method and device
CN112416884A (en) * 2020-11-23 2021-02-26 中国移动通信集团江苏有限公司 Data synchronization method and system
CN113419667A (en) * 2020-05-29 2021-09-21 阿里巴巴集团控股有限公司 Snapshot capacity calculation method and device, electronic device and computer-readable storage medium
WO2024040902A1 (en) * 2022-08-22 2024-02-29 华为云计算技术有限公司 Data access method, distributed database system and computing device cluster

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150046394A1 (en) * 2013-08-07 2015-02-12 Fujitsu Limited Storage system, storage control device, and storage medium storing control program
CN107209704A (en) * 2014-12-19 2017-09-26 甲骨文国际公司 Detect the write-in lost
WO2019120629A1 (en) * 2017-12-21 2019-06-27 Telefonaktiebolaget Lm Ericsson (Publ) On-demand snapshots from distributed data storage systems

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150046394A1 (en) * 2013-08-07 2015-02-12 Fujitsu Limited Storage system, storage control device, and storage medium storing control program
CN107209704A (en) * 2014-12-19 2017-09-26 甲骨文国际公司 Detect the write-in lost
WO2019120629A1 (en) * 2017-12-21 2019-06-27 Telefonaktiebolaget Lm Ericsson (Publ) On-demand snapshots from distributed data storage systems

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111352994A (en) * 2020-02-04 2020-06-30 浙江大华技术股份有限公司 Data synchronization method and related equipment and device
CN111352994B (en) * 2020-02-04 2023-04-18 浙江大华技术股份有限公司 Data synchronization method and related equipment and device
CN111569413A (en) * 2020-05-14 2020-08-25 北京代码乾坤科技有限公司 Data synchronization method and device
CN111569413B (en) * 2020-05-14 2023-06-13 北京代码乾坤科技有限公司 Data synchronization method and device
CN113419667A (en) * 2020-05-29 2021-09-21 阿里巴巴集团控股有限公司 Snapshot capacity calculation method and device, electronic device and computer-readable storage medium
CN112416884A (en) * 2020-11-23 2021-02-26 中国移动通信集团江苏有限公司 Data synchronization method and system
WO2024040902A1 (en) * 2022-08-22 2024-02-29 华为云计算技术有限公司 Data access method, distributed database system and computing device cluster

Also Published As

Publication number Publication date
CN110737719B (en) 2023-11-28

Similar Documents

Publication Publication Date Title
CN110737719A (en) Data synchronization method, device, equipment and computer readable storage medium
CN106874389B (en) Data migration method and device
CN107220142B (en) Method and device for executing data recovery operation
CN106100902B (en) Cloud index monitoring method and device
CN113111129B (en) Data synchronization method, device, equipment and storage medium
CN111245548A (en) Data synchronization method and device based on timestamp and computer equipment
CN114385759B (en) Configuration file synchronization method and device, computer equipment and storage medium
CN110619226A (en) Platform-based data processing method, system, equipment and storage medium
CN111680104B (en) Data synchronization method, device, computer equipment and readable storage medium
CN111737227A (en) Data modification method and system
CN110460668B (en) File uploading method and device, computer equipment and storage medium
CN109462661B (en) Data synchronization method and device, computer equipment and storage medium
CN113672692A (en) Data processing method, data processing device, computer equipment and storage medium
CN113918648A (en) Data synchronization method and device, electronic equipment and storage medium
CN113722154A (en) Data management method and system, monitoring server and storage medium
US20140222771A1 (en) Management device and management method
CN110196880B (en) Heterogeneous database data synchronization method and device, storage medium and electronic device
CN111125743B (en) Authority management method, system, computer device and computer readable storage medium
CN110990360A (en) File synchronization method based on network storage device and related components
CN115994124A (en) File scanning method, device, electronic equipment and computer readable storage medium
CN116628042A (en) Data processing method, device, equipment and medium
CN113722225A (en) Page testing method and device, computer equipment and storage medium
CN114143308A (en) File uploading information processing method and device, computer equipment and storage medium
CN112035721A (en) Crawler cluster monitoring method and device, storage medium and computer equipment
CN114547184A (en) Personnel information synchronization method, terminal device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant