CN105162855A - Incremental data synchronization method and device - Google Patents
Incremental data synchronization method and device Download PDFInfo
- Publication number
- CN105162855A CN105162855A CN201510509453.1A CN201510509453A CN105162855A CN 105162855 A CN105162855 A CN 105162855A CN 201510509453 A CN201510509453 A CN 201510509453A CN 105162855 A CN105162855 A CN 105162855A
- Authority
- CN
- China
- Prior art keywords
- data block
- cryptographic hash
- weak
- strong
- hash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an incremental data synchronization method and device. The synchronization method comprises the following steps: taking N bytes as a unit, and splitting a revision file into a plurality of data blocks; calculating to obtain a weak hash value and a strong hash value of each data block to generate a revision file hash table; and comparing an original file with the revision file hash table, obtaining the incremental data of the revision file according to a matching data block deviation value set, and carrying out incremental data synchronization. The revision file is split into the plurality of data blocks, the abstract information of each data block is calculated, then, the abstract information of the comparison data block in the original file is compared with the abstract information of each data block of the revision file, the incremental data of the revision file is calculated, and incremental data synchronization is carried out. The incremental data synchronization method has the advantages of being low in synchronization data volume, low in the calculated amount of the incremental data, low in calculation complexity and favorable in time complexity and effectively overcomes the defect that a traditional method occupies a great quantity of network resources and calculation resources.
Description
Technical field
The present invention relates to data synchronization technology, particularly a kind of incremental data synchronization method and device.
Background technology
In the information age, data are as a kind of resource, and its permanent storage, safety and efficacy, all have great importance in every field, directly affect the operational sustainability of data consumer.The loss of significant data directly will cause service disconnection, and brings huge loss.In order to improve the reliability of data, prior art adopts data synchronization technology usually, transfers data to strange land and backs up, and when certain data have active or fault, backup can be utilized to continue to provide service, or recover data.
At present, method of data synchronization relates generally to Complete Synchronization method and increment synchronization method, and Complete Synchronization method is that data are all transferred to stand-by equipment by main equipment, and increment synchronization method is main equipment by the transfer of data that changes to stand-by equipment.Actual use shows, these two kinds of synchronous method all Shortcomings.Complete Synchronization method due to transmitted data amount large, therefore Complete Synchronization process need takies a large amount of Internet resources, needs to spend the more time simultaneously, and synchronous efficiency is low.Although increment synchronization method takies a large amount of network bandwidths unlike Complete Synchronization method, but because needs do a large amount of calculating to original document and amendment file, and computation complexity is higher, therefore increment synchronization process need takies a large amount of computational resources, has a strong impact on systematic function.
Summary of the invention
In order to solve the problems of the technologies described above, the invention provides a kind of incremental data synchronization method and device, effectively overcome the defect that existing synchronous method takies a large amount of Internet resources and takies a large amount of computational resource, improve synchronous efficiency.
In order to reach the object of the invention, the invention provides a kind of incremental data synchronization method, comprising:
In units of N byte, amendment file declustering is become several data blocks;
Calculate the weak cryptographic Hash of each data block and strong cryptographic Hash, generate amendment file Hash table;
Original document and amendment file Hash table are compared, obtain according to the set of matched data block deviant the incremental data revising file, it is synchronous to carry out incremental data.
Further, described calculate each data block weak cryptographic Hash and strong cryptographic Hash be specially: weak School Affairs is carried out to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs, Hash calculation is carried out to each weak School Affairs and strong School Affairs, obtains the weak cryptographic Hash of each data block and strong cryptographic Hash.
Further, described weak School Affairs calculates and adopts adler-32 algorithm, and described strong School Affairs calculates and adopts MD5 hash algorithm.
Further, describedly original document and amendment file Hash table to be compared, obtain according to the set of matched data block deviant the incremental data revising file, it is synchronous to carry out incremental data, is specially:
Deviant M=1 is set, creates the set of matched data block deviant;
M byte is extracted to M+N-1 byte as comparison data block in original document;
Weak School Affairs and Hash calculation are carried out to described comparison data block, obtains weak cryptographic Hash; Described weak cryptographic Hash and amendment file Hash table are compared;
If mate unsuccessful, deviant M=M+1 is set, re-executes and extract comparison data block step;
If the match is successful, strong School Affairs and Hash calculation are carried out to described comparison data block, obtains strong cryptographic Hash; Described strong cryptographic Hash and amendment file Hash table are compared;
If mate unsuccessful, deviant M=M+1 is set, re-executes and extract comparison data block step;
If the match is successful, described deviant M is added in the set of described matched data block deviant, deviant M=M+N is set, re-execute and extract comparison data block step;
After all byte comparisons of original document complete, obtain revising the incremental data of file according to the set of described matched data block deviant, it is synchronous to carry out incremental data.
Further, described weak cryptographic Hash or strong cryptographic Hash are compared with amendment file Hash table and is specially: the weak cryptographic Hash of described comparison data block or strong cryptographic Hash are compared with each weak cryptographic Hash in amendment file Hash table or strong cryptographic Hash, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence and then mate unsuccessful.
In order to reach the object of the invention, present invention also offers a kind of incremental data synchronizer, comprising main equipment and stand-by equipment, described main equipment comprises fractionation module, computing module and synchronization module, wherein,
Split module, in units of N byte, amendment file declustering is become several data blocks;
Computing module, for calculating the weak cryptographic Hash of each data block and strong cryptographic Hash, generating amendment file Hash table, and described amendment file Hash table is sent to stand-by equipment;
Synchronization module, the matched data block deviant set for sending according to described stand-by equipment obtains revising the incremental data of file, and it is synchronous to carry out incremental data.
Further, described calculate each data block weak cryptographic Hash and strong cryptographic Hash be specially: weak School Affairs is carried out to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs, Hash calculation is carried out to each weak School Affairs and strong School Affairs, obtains the weak cryptographic Hash of each data block and strong cryptographic Hash.
Further, described weak School Affairs calculates and adopts adler-32 algorithm, and described strong School Affairs calculates and adopts MD5 hash algorithm.
Further, described stand-by equipment comprises extraction module, comparing module and collection modules, wherein,
Extraction module, for extracting M byte to M+N-1 byte as comparison data block in original document, described comparison data block is sent to comparing module, M is deviant;
Comparing module, is connected with the computing module of described main equipment, for receiving the amendment file Hash table that described computing module sends; Being connected with described extraction module, for calculating the weak cryptographic Hash of described comparison data block, described weak cryptographic Hash and amendment file Hash table being compared; If mate unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if the match is successful, calculate the strong cryptographic Hash of described comparison data block, described strong cryptographic Hash and amendment file Hash table are compared; If mate unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if the match is successful, described deviant is sent to collection modules, deviant is moved N number of byte backward, re-executes extraction module;
Collection modules, is connected with described comparing module, for the skew of reception being added in the set of matched data block deviant; After all byte comparisons of original document complete, the set of described matched data block deviant is sent to the synchronization module of main equipment.
Further, calculate the weak cryptographic Hash of described comparison data block or strong cryptographic Hash, described weak cryptographic Hash or strong cryptographic Hash are compared with amendment file Hash table and is specially: weak verification is carried out to described comparison data block or strong verify calculation obtains weak School Affairs or strong School Affairs, Hash calculation is carried out to described weak School Affairs or strong School Affairs and obtains the weak cryptographic Hash of comparison data block or strong cryptographic Hash, the weak cryptographic Hash of described comparison data block or strong cryptographic Hash are compared with each weak cryptographic Hash in amendment file Hash table or strong cryptographic Hash, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence then mates unsuccessful.
The invention provides a kind of incremental data synchronization method and device, first amendment Divide File is become multiple data block, and calculate the summary info of each data block, then the summary info of the summary info of comparison data block in original document with each data block of amendment file is compared, calculate the incremental data of amendment file, carry out the synchronous of incremental data, realize the data syn-chronization between main equipment and stand-by equipment.Due to only synchronous incremental data, synchrodata amount is few, and therefore the present invention program effectively overcomes the defect that existing Complete Synchronization method takies a large amount of Internet resources.Calculating due to incremental data uses efficient method of calibration, and amount of calculation is little, and computation complexity is low, computational efficiency is high, and there is good time complexity, therefore the present invention program effectively overcomes the defect that existing increment synchronization method takies a large amount of computational resource, improves synchronous efficiency.Further, the calculating of incremental data of the present invention and data content have nothing to do, and are applicable to various types of file, can be widely applied in the file synchronization of all types of file system.
Other features and advantages of the present invention will be set forth in the following description, and, partly become apparent from specification, or understand by implementing the present invention.Object of the present invention and other advantages realize by structure specifically noted in specification, claims and accompanying drawing and obtain.
Accompanying drawing explanation
Accompanying drawing is used to provide the further understanding to technical solution of the present invention, and forms a part for specification, is used from and explains technical scheme of the present invention, do not form the restriction to technical solution of the present invention with the embodiment one of the application.
Fig. 1 is the process chart of incremental data synchronization method of the present invention;
Fig. 2 is the structural representation of incremental data synchronizer of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly understand, hereinafter will be described in detail to embodiments of the invention by reference to the accompanying drawings.It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combination in any mutually.
Can perform in the computer system of such as one group of computer executable instructions in the step shown in the flow chart of accompanying drawing.Further, although show logical order in flow charts, in some cases, can be different from the step shown or described by order execution herein.
Fig. 1 is the process chart of incremental data synchronization method of the present invention.Scheme is as shown in Figure 1, with the incremental data of main equipment and stand-by equipment, technical scheme of the present invention is synchronously described.At a time, the data that main equipment stores are identical with the data that stand-by equipment stores, and are referred to as original document, and at subsequent time, the data that main equipment stores change, and are referred to as to revise file.When main equipment prepares incremental data to be synchronized to stand-by equipment, perform following steps:
Amendment file declustering, in units of N byte, is become several data blocks by step 1, main equipment.
Wherein N is positive integer, can arrange applicable N according to the length of presynchronization data.After amendment file declustering, if the length of last data block is less than N byte, be then supplemented to N byte with 0.
Step 2, weak School Affairs is carried out to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs; Hash calculation is carried out to the weak School Affairs of each data block and strong School Affairs, obtain the weak cryptographic Hash of each data block and strong cryptographic Hash, and generation amendment file Hash table sends to stand-by equipment, amendment file Hash table comprises the corresponding relation of each data block and this data block and weak School Affairs, by force School Affairs.
In actual use, weak School Affairs calculates can adopt adler-32 algorithm, and the weak School Affairs calculated is the informative abstract of each data block, is the character string of 32.The object of weak School Affairs is adopted to be the feature utilizing its comparison efficiency high.Strong School Affairs calculates can adopt MD5 hash algorithm, and the strong School Affairs calculated is the informative abstract of each data block equally, is the character string of 128.The object of strong School Affairs is adopted to be the deficiency easily collided when comparing to make up weak School Affairs.The amendment file Hash table generated is as shown in the table:
| Data block 1 | Weak cryptographic Hash X1 | Strong cryptographic Hash Y1 |
| Data block 2 | Weak cryptographic Hash X2 | Strong cryptographic Hash Y2 |
| Data block 3 | Weak cryptographic Hash X3 | Strong cryptographic Hash Y3 |
| ...... | ...... | ...... |
The present invention weak School Affairs and strong School Affairs calculate cryptographic Hash, leave in Hash list, so just obtain the search performance that time complexity is O (1).In actual use, the weak cryptographic Hash of each data block can use the weak School Affairs of 32 as entrance, and the weak School Affairs of 16 also can be used as entrance.
Step 3, stand-by equipment receive the amendment file Hash table that main equipment sends, and preserve.
Step 4, stand-by equipment arrange deviant M=1, create the set of matched data block deviant.
Step 5, stand-by equipment extract M byte to M+N-1 byte as comparison data block in original document.
Step 6, weak School Affairs calculating is carried out to this comparison data block, Hash calculation is carried out to the weak School Affairs obtained, obtains weak cryptographic Hash; This weak cryptographic Hash and amendment file Hash table are compared, if the match is successful, performs step 7, if mate unsuccessful, perform step 11.
Step 7, strong School Affairs calculating is carried out to this comparison data block, Hash calculation is carried out to the strong School Affairs obtained, obtains strong cryptographic Hash; This strong cryptographic Hash and amendment file Hash table are compared, if the match is successful, performs step 8, if mate unsuccessful, perform step 11.
Wherein, weak cryptographic Hash or strong cryptographic Hash are compared with amendment file Hash table, refer to and the weak cryptographic Hash of comparison data block or strong cryptographic Hash are compared with each weak cryptographic Hash in amendment file Hash table or strong cryptographic Hash, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence and then mate unsuccessful.
Step 8, the deviant M of this comparison data block to be added in the set of matched data block deviant.
Step 9, judge whether the length of original document is greater than M+N byte, is perform step 10, otherwise perform step 13.
Step 10, deviant M=M+N is set, performs step 5.
Step 11, judge whether the length of original document is greater than M+1 byte, is perform step 12, otherwise perform step 13.
Step 12, deviant M=M+1 is set, performs step 5.
The set of matched data block deviant is sent to main equipment by step 13, stand-by equipment.
Step 14, main equipment, to matched data block deviant set supplement collection, obtain amendment file and the unmatched data block of original document (incremental data), unmatched data block are synchronized to stand-by equipment, terminate.
Aforementioned processing flow process is actually the technical scheme of being compared with amendment file by the comparison data block of original document one by one, and the former data block of each comparison is equally in units of N number of byte.When starting comparison, first extract original document top n byte as comparison data block, namely the 1st byte is to N byte (deviant M=1).Weak School Affairs calculating is carried out to this comparison data block, Hash calculation is carried out to the weak School Affairs calculated, obtain the weak cryptographic Hash of this comparison data block, the weak cryptographic Hash of this comparison data block and each weak cryptographic Hash in amendment file Hash table are compared.
If weak cryptographic Hash comparison result there is occurrence, then the match is successful, then calculates the strong cryptographic Hash of this comparison data block, this strong cryptographic Hash and each strong cryptographic Hash in amendment file Hash table compared.If strong cryptographic Hash comparison result there is occurrence, then the match is successful, illustrate in amendment file to there is the data block identical with this comparison data block, then record this position of comparison data block in original document, deviant M by this comparison data block adds in the set of matched data block deviant, deviant M is moved backward N number of byte (deviant=M+N), re-start and extract comparison data block, weak cryptographic Hash calculates and the processing procedure of comparison.
If weak cryptographic Hash comparison result there is not occurrence, then mate unsuccessful, data block not identical with this comparison data block in amendment file is described, then without the need to carrying out follow-up comparison, by a deviant M mobile byte backward, namely extraction original document the 2nd byte is to N+1 byte as new comparison data block, re-starts the processing procedure of the calculating of weak cryptographic Hash and comparison.
If strong cryptographic Hash comparison result there is not occurrence, then mate unsuccessful, data block not identical with this comparison data block in amendment file is described, then by a comparison data block mobile byte backward, re-starts and extract comparison data block, weak cryptographic Hash calculates and the processing procedure of comparison.
After all byte comparisons of original document complete by stand-by equipment, the set of matched data block deviant is sent to main equipment, main equipment is to matched data block deviant set supplement collection, just can obtain revising the file byte not identical with original document, these not identical bytes are formed unmatched data block (i.e. incremental data), unmatched data block is synchronized in stand-by equipment, namely completes main equipment synchronous with the incremental data of stand-by equipment.Equally, if the length of comparison data block is less than N byte, then N byte is supplemented to 0.
The invention provides a kind of incremental data synchronization method, first amendment Divide File is become multiple data block, and calculate the summary info of each data block, then the summary info of the summary info of comparison data block in original document with each data block of amendment file is compared, calculate the incremental data of amendment file, carry out the synchronous of incremental data, realize the data syn-chronization between main equipment and stand-by equipment.Due to only synchronous incremental data, synchrodata amount is few, and therefore the present invention program effectively overcomes the defect that existing Complete Synchronization method takies a large amount of Internet resources.Calculating due to incremental data uses efficient method of calibration, and amount of calculation is little, and computation complexity is low, computational efficiency is high, and there is good time complexity, therefore the present invention program effectively overcomes the defect that existing increment synchronization method takies a large amount of computational resource, improves synchronous efficiency.Further, the calculating of incremental data of the present invention and data content have nothing to do, and are applicable to various types of file, can be widely applied in the file synchronization of all types of file system.
On aforementioned incremental data synchronization method technical scheme basis, present invention also offers a kind of incremental data synchronizer.Fig. 2 is the structural representation of incremental data synchronizer of the present invention, as shown in Figure 2, incremental data synchronizer of the present invention comprises main equipment and stand-by equipment, the file that main equipment stores is amendment file, the file that stand-by equipment stores is original document, and main equipment comprises fractionation module, computing module and synchronization module, and stand-by equipment comprises extraction module, comparing module and collection modules, wherein
Split module, in units of N byte, amendment file declustering is become several data blocks, and several data blocks are sent to computing module.
Computing module, is connected with the comparing module of described fractionation module and stand-by equipment, for calculating the weak cryptographic Hash of each data block and strong cryptographic Hash, generating amendment file Hash table, and described amendment file Hash table being sent to the comparing module of stand-by equipment.
Synchronization module, is connected with the collection modules of described stand-by equipment, and the matched data block deviant set for sending according to the collection modules of described stand-by equipment obtains revising the incremental data of file, and it is synchronous to carry out incremental data.
Extraction module, for extracting M byte to M+N-1 byte as comparison data block in original document, described comparison data block is sent to comparing module, M is deviant.
Comparing module, is connected with the computing module of described main equipment, for receiving the amendment file Hash table that described computing module sends; Being connected with described extraction module, for receiving the comparison data block that described extraction module sends, calculating the weak cryptographic Hash of described comparison data block, described weak cryptographic Hash and amendment file Hash table are compared; If described weak Hash values match is unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if described weak Hash values match success, calculate the strong cryptographic Hash of described comparison data block, described strong cryptographic Hash and amendment file Hash table are compared; If described strong Hash values match is unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if described strong Hash values match success, described deviant is sent to collection modules, deviant is moved N number of byte backward, re-executes extraction module.
Collection modules, is connected with described comparing module, for receiving the deviant that described comparing module sends, and described deviant is added in the set of matched data block deviant; Be connected with the synchronization module of described main equipment, after all byte comparisons of original document complete, the set of described matched data block deviant sent to the synchronization module of main equipment.
In such scheme, the weak cryptographic Hash and the strong cryptographic Hash that calculate each data block are specially: carry out weak School Affairs to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs, Hash calculation is carried out to each weak School Affairs and strong School Affairs, obtains the weak cryptographic Hash of each data block and strong cryptographic Hash.Described weak School Affairs calculates and adopts adler-32 algorithm, and described strong School Affairs calculates and adopts MD5 hash algorithm.
Wherein, calculate the weak cryptographic Hash of comparison data block, weak cryptographic Hash is compared with amendment file Hash table and is specially: weak verify calculation is carried out to comparison data block and obtains weak School Affairs, the weak cryptographic Hash that Hash calculation obtains comparison data block is carried out to weak School Affairs, the weak cryptographic Hash of comparison data block and each weak cryptographic Hash in amendment file Hash table are compared, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence and then mate unsuccessful.Calculate the strong cryptographic Hash of comparison data block, strong cryptographic Hash is compared with amendment file Hash table and is specially: strong verify calculation is carried out to comparison data block and obtains strong School Affairs, the strong cryptographic Hash that Hash calculation obtains comparison data block is carried out to strong School Affairs, the strong cryptographic Hash of comparison data block and each strong cryptographic Hash in amendment file Hash table are compared, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence and then mate unsuccessful.
Although the execution mode disclosed by the present invention is as above, the execution mode that described content only adopts for ease of understanding the present invention, and be not used to limit the present invention.Those of skill in the art belonging to any the present invention; under the prerequisite not departing from the spirit and scope disclosed by the present invention; any amendment and change can be carried out in the form implemented and details; but scope of patent protection of the present invention, the scope that still must define with appending claims is as the criterion.
Claims (10)
1. an incremental data synchronization method, is characterized in that, comprising:
In units of N byte, amendment file declustering is become several data blocks;
Calculate the weak cryptographic Hash of each data block and strong cryptographic Hash, generate amendment file Hash table;
Original document and amendment file Hash table are compared, obtain according to the set of matched data block deviant the incremental data revising file, it is synchronous to carry out incremental data.
2. method according to claim 1, it is characterized in that, described calculate each data block weak cryptographic Hash and strong cryptographic Hash be specially: weak School Affairs is carried out to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs, Hash calculation is carried out to each weak School Affairs and strong School Affairs, obtains the weak cryptographic Hash of each data block and strong cryptographic Hash.
3. method according to claim 2, is characterized in that, described weak School Affairs calculates and adopts adler-32 algorithm, and described strong School Affairs calculates and adopts MD5 hash algorithm.
4. according to the arbitrary described method of claims 1 to 3, it is characterized in that, describedly original document and amendment file Hash table to be compared, obtain according to the set of matched data block deviant the incremental data revising file, it is synchronous to carry out incremental data, is specially:
Deviant M=1 is set, creates the set of matched data block deviant;
M byte is extracted to M+N-1 byte as comparison data block in original document;
Weak School Affairs and Hash calculation are carried out to described comparison data block, obtains weak cryptographic Hash; Described weak cryptographic Hash and amendment file Hash table are compared;
If mate unsuccessful, deviant M=M+1 is set, re-executes and extract comparison data block step;
If the match is successful, strong School Affairs and Hash calculation are carried out to described comparison data block, obtains strong cryptographic Hash; Described strong cryptographic Hash and amendment file Hash table are compared;
If mate unsuccessful, deviant M=M+1 is set, re-executes and extract comparison data block step;
If the match is successful, described deviant M is added in the set of described matched data block deviant, deviant M=M+N is set, re-execute and extract comparison data block step;
After all byte comparisons of original document complete, obtain revising the incremental data of file according to the set of described matched data block deviant, it is synchronous to carry out incremental data.
5. method according to claim 4, is characterized in that,
Described weak cryptographic Hash or strong cryptographic Hash are compared with amendment file Hash table and is specially: the weak cryptographic Hash of described comparison data block or strong cryptographic Hash are compared with each weak cryptographic Hash in amendment file Hash table or strong cryptographic Hash, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence and then mate unsuccessful.
6. an incremental data synchronizer, comprises main equipment and stand-by equipment, it is characterized in that, described main equipment comprises fractionation module, computing module and synchronization module, wherein,
Split module, in units of N byte, amendment file declustering is become several data blocks;
Computing module, for calculating the weak cryptographic Hash of each data block and strong cryptographic Hash, generating amendment file Hash table, and described amendment file Hash table is sent to stand-by equipment;
Synchronization module, the matched data block deviant set for sending according to described stand-by equipment obtains revising the incremental data of file, and it is synchronous to carry out incremental data.
7. device according to claim 6, it is characterized in that, described calculate each data block weak cryptographic Hash and strong cryptographic Hash be specially: weak School Affairs is carried out to each data block and strong School Affairs calculates, obtain the weak School Affairs of each data block and strong School Affairs, Hash calculation is carried out to each weak School Affairs and strong School Affairs, obtains the weak cryptographic Hash of each data block and strong cryptographic Hash.
8. device according to claim 7, is characterized in that, described weak School Affairs calculates and adopts adler-32 algorithm, and described strong School Affairs calculates and adopts MD5 hash algorithm.
9., according to the arbitrary described device of claim 6 ~ 8, it is characterized in that, described stand-by equipment comprises extraction module, comparing module and collection modules, wherein,
Extraction module, for extracting M byte to M+N-1 byte as comparison data block in original document, described comparison data block is sent to comparing module, M is deviant;
Comparing module, is connected with the computing module of described main equipment, for receiving the amendment file Hash table that described computing module sends; Being connected with described extraction module, for calculating the weak cryptographic Hash of described comparison data block, described weak cryptographic Hash and amendment file Hash table being compared; If mate unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if the match is successful, calculate the strong cryptographic Hash of described comparison data block, described strong cryptographic Hash and amendment file Hash table are compared; If mate unsuccessful, by a deviant M mobile byte backward, re-execute extraction module, if the match is successful, described deviant is sent to collection modules, deviant is moved N number of byte backward, re-executes extraction module;
Collection modules, is connected with described comparing module, for the skew of reception being added in the set of matched data block deviant; After all byte comparisons of original document complete, the set of described matched data block deviant is sent to the synchronization module of main equipment.
10. device according to claim 9, is characterized in that,
Calculate the weak cryptographic Hash of described comparison data block or strong cryptographic Hash, described weak cryptographic Hash or strong cryptographic Hash are compared with amendment file Hash table and is specially: weak verification is carried out to described comparison data block or strong verify calculation obtains weak School Affairs or strong School Affairs, Hash calculation is carried out to described weak School Affairs or strong School Affairs and obtains the weak cryptographic Hash of comparison data block or strong cryptographic Hash, the weak cryptographic Hash of described comparison data block or strong cryptographic Hash are compared with each weak cryptographic Hash in amendment file Hash table or strong cryptographic Hash, judge whether to there is occurrence, then the match is successful to there is occurrence, there is not occurrence then mates unsuccessful.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510509453.1A CN105162855A (en) | 2015-08-18 | 2015-08-18 | Incremental data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510509453.1A CN105162855A (en) | 2015-08-18 | 2015-08-18 | Incremental data synchronization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105162855A true CN105162855A (en) | 2015-12-16 |
Family
ID=54803608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510509453.1A Pending CN105162855A (en) | 2015-08-18 | 2015-08-18 | Incremental data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105162855A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106227901A (en) * | 2016-09-19 | 2016-12-14 | 郑州云海信息技术有限公司 | A kind of based on heavily deleting and compressing parallel space method for saving |
| CN107346271A (en) * | 2016-05-05 | 2017-11-14 | 华为技术有限公司 | The method and calamity of Backup Data block are for end equipment |
| CN107835206A (en) * | 2017-09-19 | 2018-03-23 | 平安普惠企业管理有限公司 | Data-updating method, server and computer-readable recording medium |
| WO2018054497A1 (en) * | 2016-09-23 | 2018-03-29 | Huawei Technologies Co., Ltd. | Method and device for synchronising data between devices |
| WO2018177190A1 (en) * | 2017-03-28 | 2018-10-04 | 上海瑞麒维网络科技有限公司 | Method and device for synchronizing blockchain data |
| WO2019071801A1 (en) * | 2017-10-10 | 2019-04-18 | 语联网（武汉）信息技术有限公司 | Data synchronization method |
| CN110348732A (en) * | 2019-07-08 | 2019-10-18 | 泰华智慧产业集团股份有限公司 | Enterprise's representation data preprocess method and system based on hash algorithm |
| CN110599169A (en) * | 2019-09-16 | 2019-12-20 | 腾讯科技（深圳）有限公司 | Data processing method, device, terminal and medium |
| CN110837537A (en) * | 2019-10-12 | 2020-02-25 | 平安银行股份有限公司 | Order data synchronization method and device, computer readable storage medium and server |
| CN111125258A (en) * | 2019-12-26 | 2020-05-08 | 哈尔滨工业大学（深圳） | Data synchronization method, client, server and system |
| CN111294391A (en) * | 2020-01-17 | 2020-06-16 | 深信服科技股份有限公司 | Configuration synchronization method, device, equipment and readable storage medium |
| CN111569413A (en) * | 2020-05-14 | 2020-08-25 | 北京代码乾坤科技有限公司 | Data synchronization method and device |
| CN111581031A (en) * | 2020-05-13 | 2020-08-25 | 上海英方软件股份有限公司 | Data synchronization method and device based on RDC (remote data center) indefinite-length partitioning strategy |
| CN112765276A (en) * | 2021-01-27 | 2021-05-07 | 云账户技术(天津)有限公司 | Incremental data synchronization method and device |
| WO2021114025A1 (en) * | 2019-12-09 | 2021-06-17 | 深圳市欢太科技有限公司 | Incremental data determination method, incremental data determination apparatus, server and terminal device |
| CN115145943A (en) * | 2022-09-06 | 2022-10-04 | 北京麦聪软件有限公司 | Multi-data-source metadata rapid comparison method, system, device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102065098A (en) * | 2010-12-31 | 2011-05-18 | 网宿科技股份有限公司 | Method and system for synchronizing data among network nodes |
| CN103379160A (en) * | 2012-04-25 | 2013-10-30 | 上海咏云信息技术有限公司 | Difference synchronizing method for oversized file |
| CN103428242A (en) * | 2012-05-18 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method, device and system for increment synchronization |
| CN104023085A (en) * | 2014-06-25 | 2014-09-03 | 武汉大学 | Security cloud storage system based on increment synchronization |
| CN104348884A (en) * | 2013-08-08 | 2015-02-11 | 中国科学院计算机网络信息中心 | Cloud storage automatic synchronization method |
| US20150220747A1 (en) * | 2006-05-05 | 2015-08-06 | Hybir Inc. | Group based complete and incremental computer file backup system, process and apparatus |
-
2015
- 2015-08-18 CN CN201510509453.1A patent/CN105162855A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150220747A1 (en) * | 2006-05-05 | 2015-08-06 | Hybir Inc. | Group based complete and incremental computer file backup system, process and apparatus |
| CN102065098A (en) * | 2010-12-31 | 2011-05-18 | 网宿科技股份有限公司 | Method and system for synchronizing data among network nodes |
| CN103379160A (en) * | 2012-04-25 | 2013-10-30 | 上海咏云信息技术有限公司 | Difference synchronizing method for oversized file |
| CN103428242A (en) * | 2012-05-18 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method, device and system for increment synchronization |
| CN104348884A (en) * | 2013-08-08 | 2015-02-11 | 中国科学院计算机网络信息中心 | Cloud storage automatic synchronization method |
| CN104023085A (en) * | 2014-06-25 | 2014-09-03 | 武汉大学 | Security cloud storage system based on increment synchronization |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107346271A (en) * | 2016-05-05 | 2017-11-14 | 华为技术有限公司 | The method and calamity of Backup Data block are for end equipment |
| CN106227901A (en) * | 2016-09-19 | 2016-12-14 | 郑州云海信息技术有限公司 | A kind of based on heavily deleting and compressing parallel space method for saving |
| WO2018054497A1 (en) * | 2016-09-23 | 2018-03-29 | Huawei Technologies Co., Ltd. | Method and device for synchronising data between devices |
| WO2018177190A1 (en) * | 2017-03-28 | 2018-10-04 | 上海瑞麒维网络科技有限公司 | Method and device for synchronizing blockchain data |
| CN107835206A (en) * | 2017-09-19 | 2018-03-23 | 平安普惠企业管理有限公司 | Data-updating method, server and computer-readable recording medium |
| WO2019071801A1 (en) * | 2017-10-10 | 2019-04-18 | 语联网（武汉）信息技术有限公司 | Data synchronization method |
| CN110348732A (en) * | 2019-07-08 | 2019-10-18 | 泰华智慧产业集团股份有限公司 | Enterprise's representation data preprocess method and system based on hash algorithm |
| CN110599169B (en) * | 2019-09-16 | 2021-03-26 | 腾讯科技（深圳）有限公司 | Data processing method, device, terminal and medium |
| CN110599169A (en) * | 2019-09-16 | 2019-12-20 | 腾讯科技（深圳）有限公司 | Data processing method, device, terminal and medium |
| CN110837537A (en) * | 2019-10-12 | 2020-02-25 | 平安银行股份有限公司 | Order data synchronization method and device, computer readable storage medium and server |
| CN110837537B (en) * | 2019-10-12 | 2024-05-31 | 平安银行股份有限公司 | Order data synchronization method, order data synchronization device, computer readable storage medium and server |
| WO2021114025A1 (en) * | 2019-12-09 | 2021-06-17 | 深圳市欢太科技有限公司 | Incremental data determination method, incremental data determination apparatus, server and terminal device |
| CN111125258B (en) * | 2019-12-26 | 2023-03-28 | 哈尔滨工业大学（深圳） | Data synchronization method, client, server and system |
| CN111125258A (en) * | 2019-12-26 | 2020-05-08 | 哈尔滨工业大学（深圳） | Data synchronization method, client, server and system |
| CN111294391A (en) * | 2020-01-17 | 2020-06-16 | 深信服科技股份有限公司 | Configuration synchronization method, device, equipment and readable storage medium |
| CN111581031A (en) * | 2020-05-13 | 2020-08-25 | 上海英方软件股份有限公司 | Data synchronization method and device based on RDC (remote data center) indefinite-length partitioning strategy |
| CN111569413A (en) * | 2020-05-14 | 2020-08-25 | 北京代码乾坤科技有限公司 | Data synchronization method and device |
| CN111569413B (en) * | 2020-05-14 | 2023-06-13 | 北京代码乾坤科技有限公司 | Data synchronization method and device |
| CN112765276A (en) * | 2021-01-27 | 2021-05-07 | 云账户技术(天津)有限公司 | Incremental data synchronization method and device |
| CN115145943A (en) * | 2022-09-06 | 2022-10-04 | 北京麦聪软件有限公司 | Multi-data-source metadata rapid comparison method, system, device and storage medium |
| CN115145943B (en) * | 2022-09-06 | 2023-02-28 | 北京麦聪软件有限公司 | Method, system, equipment and storage medium for rapidly comparing metadata of multiple data sources |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105162855A (en) | Incremental data synchronization method and device | |
| CN102684827B (en) | Data processing method and data processing equipment | |
| CN103778136A (en) | Cross-room database synchronization method and system | |
| CN103218398B (en) | Intelligent substation SCL (substation configuration description language) file difference comparison method | |
| CN101673289B (en) | Method and device for constructing distributed file storage framework | |
| CN102946410A (en) | Method and device for network synchronization | |
| CN102946411A (en) | Network synchronization system | |
| CN106250444A (en) | The real-time Input System of a kind of heterogeneous data source and method | |
| CN104361068B (en) | Parallel method of partition and system during a kind of data deduplication | |
| CN107037978A (en) | Data Migration bearing calibration and system | |
| CN103106200B (en) | Non-relational database synchronization system and double write synchronous method | |
| CN104243508A (en) | Server, client side and file synchronization method | |
| CN104954101B (en) | Multiple terminals method of data synchronization based on synchronous ACK | |
| CN103885829A (en) | Virtual machine cross-data-center dynamic migration optimization method based on statistics | |
| CN103716056B (en) | Data compression method, uncompressing data and equipment | |
| CN104484242A (en) | Method for breaking-point transmission in copy-on-write snapshot backup process | |
| CN104219298A (en) | Cluster system and data backup method thereof | |
| CN105450712A (en) | Data transmission method and device | |
| MX2021006767A (en) | Data transmission of value pair in network systems. | |
| CN112395360B (en) | Data synchronization method, device, apparatus and medium based on non-relational database | |
| CN108694205B (en) | Method and device for matching target field | |
| CN111078463A (en) | Data backup method, device and system | |
| Sari et al. | The implementation of timestamp, bitmap and rake algorithm on data compression and data transmission from iot to cloud | |
| CN110196880B (en) | Heterogeneous database data synchronization method and device, storage medium and electronic device | |
| CN103106144B (en) | A kind of internal memory index compression method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination |