CN101354708B - Remote file rapid synchronization method - Google Patents

Abstract

The invention provides a method for rapidly synchronizing a remote file, belonging to the technical field of a computer network. The method stores new and old versions of a set of files at the client through the snapshot technique, avoids the data request on a server in differential calculation, and copies and moves the file at the client; and the server repeats the operation and does not transmit the file content. The method has the characteristics that the differential calculation is rapid, the network communication volume is small, and the differential replay is rapid, accelerates the file synchronization between the client and the server, and has wide potential application in the aspects such as the file backup and file recovery, and the Web cache.

Description

A kind of remote file rapid synchronization method

One, technical field

The present invention proposes a kind of remote file rapid synchronization method, belongs to technical field of the computer network.

Two, background technology

Telefile is widely used at aspects such as file backup and recovery, Web buffer memorys synchronously.Telefile stationary problem key is: the file status of realizing server and client as how minimum Internet traffic is synchronous, particularly between the file of preserving on the file of preserving on the server and the client difference than hour, synchronous speed should be very fast.Operations such as the difference on the client and server between the file can be by document creation, duplicate, deletion, rename, content modification cause, existing telefile method for synchronous, rsync for example is not even if file changes, when difference is calculated, also to produce network traffics; And when handling by file copy or moving the difference of introducing, the file transfer of duplicating or move operation produces is arrived server, cause huge Internet traffic.

The described file synchronisation method of patent publication No. CN1549130 simply uses copy and overlapping operation to finish the synchronous of file, can't avoid the Network Transmission of mass data.

The present invention proposes a kind of remote file rapid synchronization method, has following characteristics:

1) difference between client calculating new and old edition file, spanned file state difference collection, the difference computing velocity is fast, and difference is calculated and is not produced network traffics;

2) for the file copy or the move operation of client, server this operation of resetting, transfer files content not, the difference playback speed is fast.

Three, summary of the invention

The present invention proposes a kind of remote file rapid synchronization method.Its basic thought is: use the new and old edition of snapping technique at one group of file of client storage, difference between client calculation document new and old edition, spanned file state difference collection, the file difference that operations such as unified description document content change and rename are introduced, wherein rename comprises file copy, the complete trails filename that causes such as moves and change, file status difference collection arrives server by Network Transmission, according to the file difference collection, the file operation of server playback client realizes that client is synchronous fast to the file of server.

Before elaborating the present invention, do following term definition earlier:

(1) file f _i, refer to that the synchronous file (directory name is expressed as file) of needs comprises absolute path and filename.

(2) file set F refers to the set that the synchronous file of a plurality of needs is formed: F={f ₁, f ₂..., f _i..., f _n, at t ₁Constantly be designated as F ₁, at t ₂Constantly be designated as F ₂, t wherein ₁＜t ₂

(3) file difference collection Δ F=(F ₁-F ₂) ∪ (F ₂-F ₁), F wherein ₁, F ₂Be t ₁And t ₂File set constantly.When file set does not have difference, when promptly needing synchronous file and catalogue not to increase and decrease, Δ F=Φ then.

(4) file status s _i ^t＜f _i ^p, f _i ^c, a file comprises absolute path and filename f at t state constantly _i ^p, file content f _i ^c

(5) file status collection S ^t, refer to that one group of file at t state constantly, can be expressed as $S^t=\{s_1^t,s_2^t,...,s_i^t,...,s_n^t\}.$

(6) file status discrepancy delta s _i＜Δ f _i ^p, Δ f _i ^c, refer to file set F ₁And F ₂In a file at t ₁And t ₂State variation constantly is expressed as $\mathrm{\&Delta;}{\mathrm{<figure-callout\; id="2"\; label="s\; i"\; filenames="H200810045679001E00011.png,H200810045679001E00031.png"\; state="\{\{state\}\}">s</figure-callout>}}_i^{}=s_i^{t_2}-s_i^{t_1}.$ File t ₂State can be by t constantly ₁State and file status difference calculate constantly, are expressed as $s_i^{{\mathrm{<figure-callout\; id="2"\; label="t"\; filenames="H200810045679001E00011.png,H200810045679001E00031.png"\; state="\{\{state\}\}">t</figure-callout>}}_2}=\mathrm{\&Delta;}{\mathrm{<figure-callout\; id="2"\; label="s\; i"\; filenames="H200810045679001E00011.png,H200810045679001E00031.png"\; state="\{\{state\}\}">s</figure-callout>}}_i^{}+s_i^{t_1}.$

(7) file status difference collection Δ S is by file set F ₁And F ₂In one or one group of file status discrepancy delta s _iThe set Δ S={ Δ s that forms ₁, Δ s ₂..., Δ s _i..., Δ s _n.

(8) the identical file of search file content $F_R=O_c(f_i^c,f_k^c,F^{\&prime;}),$ Comparison document f _iWith each f _kWhether ∈ F ' file content is identical, if identical, export file name f _k ^pThe set F that forms _R, otherwise F _R=Φ.

(9) judge whether to have same file name $f^P=O_p(f_i^p,F^{\&prime;\&prime;}),$ Judge file f _iFilename f _i ^pWhether belong to F ".If $f_i^p\&Element;F^{\&prime;\&prime;},$ Then output $f^p=f_i^p,$ Otherwise output is empty.

Four, description of drawings

Fig. 1 is a fundamental diagram of the present invention.

Fig. 2 is the step of the identical file of search file content.

Fig. 3 is the step of spanned file state difference collection.

Fig. 4 is the step of storage of file status difference collection and transmission.

Fig. 5 is the step that file status difference collection is reset.

Five, embodiment

Describe concrete grammar of the present invention in detail below in conjunction with accompanying drawing.

Fig. 1 is a fundamental diagram of the present invention.

Fig. 1 is a fundamental diagram of the present invention, among the figure

The file at expression arrow two ends has identical filename, but file content there are differences, corresponding to the file content retouching operation of client.The file of " → " expression arrow the right and left has identical file content, but filename is inconsistent, corresponding to the file rename operation of client.The file on " ← " expression arrow the right identical file of filename perhaps in the arrow left side does not exist with it is corresponding to the file creation operation of client.

The file on the expression arrow left side identical file of filename perhaps in arrow the right does not exist with it is corresponding to the file delete operation of client.T among the figure ₁The state consistency of file on the file of client snapshot description constantly and the server is corresponding to file set F ₁, with t ₂Snapshot is represented the current file status of client constantly, corresponding to file set F ₂, t wherein ₁＜t ₂Calculate these two the file set F that snapshot write down constantly ₂And F ₁State difference collection Δ S, server is finished file synchronization according to Δ S.

Particularly, the concrete steps of a kind of remote file rapid synchronization method of the present invention's proposition may further comprise the steps:

1) step of the file that the search file content is identical;

2) step of spanned file state difference collection;

3) step of storage of file status difference collection and transmission;

4) step of file status difference collection playback.

Fig. 2 is the step of the identical file of search file content.

Fig. 2 has provided the identical file of search file content $F_R=O_c(f_i^c,f_k^c,F^{\&prime;})$ Concrete computing method.All Files among the file set F ' is carried out following operation, judge f _iWhether and the file f among the F ' _kThe content unanimity, promptly $f_i^c=f_k^c,$ If the content unanimity is then with f _kFilename f _k ^pAdd set F _RIn, f wherein _i ^cCan be expressed as＜f _i ^l, f _i ^t, f _i ^h, f _i ^lBe file size; f _i ^tBe the last modification time of file, f _i ^hBe the file hash value.Concrete steps are as follows:

1) step of comparison document length: compare f _i ^lWhether equal f _k ^lIf, not etc., f then _i ^cAnd f _k ^cInconsistent, change 4);

2) step of comparison document hashed value: compare f _i ^hWith f _k ^hWhether identical, if difference, then f _i ^cAnd f _k ^cInconsistent, change 4);

3) filename that will have a same file content deposits the step of result set in: through 2 steps in front, judge f _i ^cAnd f _k ^cUnanimity is with f _k ^pAdd F _R

4) judge the step that traversal finishes: check F _RWhether middle All Files travels through is finished, if traversal is finished, then withdraw from,

Otherwise change 1)

Fig. 3 is the step of spanned file state difference collection.

Fig. 3 has provided the concrete grammar of spanned file state difference collection.Specific t ₁, t ₂File set F constantly ₁, F ₂, to All Files f _i∈ F ₂, the state difference Δ s of calculating this document _i, all Δ s form Δ S, and concrete steps are as follows:

1) step of the file that the search file content is identical: executable operations $F_R=O_c(f_i^c,f_k^c,F_1),$ The result is designated as F _cIf, F _cBe not empty set, then change 2), otherwise change 3);

2) detect the step that the file rename is operated: executable operations $f^p=O_p(f_i^p,F_c),$ Judge f _i ^pWhether belong to F _c, if $f_i^p\&Element;F_c,$ File status no change then, if $f_i^p\&NotElement;F_c,$ Then file produces by duplicating, moving, and generates $\mathrm{\&Delta;}{s}_i=\mathrm{\&Delta;}{f}_i^p=<f_k^p,f_i^p>,$ f _k ^pThe expression source filename, f _i ^pExpression file destination name is with Δ s _iAdd among the Δ S;

3) step of the new file of detection: executable operations $f^p=O_p(f_i^p,F_1),$ Judge f _i ^pWhether belong to F ₁If, $f^p=f_i^p,$ Then the supporting paper name does not become, f _iFile content changes, and changes 4), otherwise f _iBe new file, Δ s _i=Δ f _i, with Δ s _iAdd among the Δ S;

4) step of calculation document content difference: by classical rsync algorithm computation $\mathrm{\&Delta;}{f}_i^c=\mathrm{rsync}(f_i^c,f_j^c),,$ F wherein _i ^cBe f _iNew file content, and f _j ^cBe f _iThe ancient deed content, promptly $\mathrm{\&Delta;}{s}_i=\mathrm{\&Delta;}{f}_i^c,$ With Δ s _iAdd among the Δ S.

Fig. 4 is the step of storage of file status difference collection and transmission.

Fig. 4 has provided the concrete grammar of storage of file difference collection and transmission.With file FS _r, FS _c, FS _n, FS _dDeposit difference dissimilar among the Δ S respectively, to depositing four class files of difference among the Δ S, preface: FS successively _r→ FS _c→ FS _n→ FS _d, respectively with File Open, and with file content by Network Transmission to remote server.Concrete steps are as follows:

1) step that stores classifiedly of file status difference collection: the file status difference that writes down among the Δ S is existed respectively in the different files of 4 classes, wherein FS _rPreserve the identical but difference of route diversion of file content, promptly $\mathrm{\&Delta;}{s}_i=\mathrm{\&Delta;}{f}_i^p;$ FS _cPreserve filename and difference content that file content difference takes place, promptly ${\mathrm{\&Delta;s}}_i=\mathrm{\&Delta;}{f}_i^c;$ FS _nPreserve the filename and the file content of newly-increased file, i.e. Δ s _i=f _iFS _dExpression deleted file, i.e. Δ F '=F ₁-F ₂, every record r _dFilename for deleted file;

2) open the step of the file of depositing file status difference: preface FS successively _r→ FS _c→ FS _n→ FS _d, open various types of difference state storage files respectively;

3) content that opens file the step of transmission file status difference: with 2) sends to remote server by network;

4) check that file status difference sends the step that finishes: checking whether all the difference storage file has sent finishes, and finishes if the result sends, and then withdraws from, otherwise switches to next difference storage file, changes 2).

Fig. 5 is the step that file status difference collection is reset.

Fig. 5 has provided the concrete grammar that file status difference collection is reset.Server is after receiving the file status difference collection that client is come by Network Transmission, and playback file status difference collection is promptly carried out S ²=Δ S+S ¹, finish the synchronous of file, playback order is: FS _r→ FS _c→ FS _n→ FS _dConcrete steps are as follows:

1) step of playback client file rename operation: according to FS _rThe file rename service data of middle record, at first spanned file rename playback script is carried out script then, and concrete steps are as follows:

1. the step of all records of preprocessed file rename: establish FS _rThe set that all records are formed in the file is R, and every is recorded as r＜r _s, r _t, wherein $r_s=f_k^p$ Be source filename, $r_t=f_i^p$ For the file destination name, according to r＜r _s, r _t, generate source file destination name r _sSet F _s, file destination name r _tSet F _t, calculate F _tIn each r _tPathdepth, and according to the pathdepth descending sort;

2. the step of spanned file rename playback script: state 3 diques, represent playback FS respectively _rIn the operation that need carry out of file difference, resolve FS then _rIn every record, and structure file movement or replicate run be in corresponding formation, the operation of preserving in final 3 formations is the playback script of being asked, concrete steps are as follows:

(a) step of statement dique: statement dique Q _CP, Q _MVAnd Q _RD, Q wherein _CPPreserve replicate run, Q _MVPreserve the operation of rename operation and filename exchange first half; Q _RDPreserve the operation of execute file name exchange latter half;

(b) detect the step of filename swap operation: for according to r _tEach record r＜r of descending sort _s, r _tIn r _s, at set F _tIf middle search is r _s∈ F _tThe swap file name then taking place, utilize random number generation function RD, system clock T to generate unique filename, produces rename operation script MV (r _s, RD (T)) and be inserted into Q _MVQueue heads produces script MV (RD (T), r _t) be inserted into Q _RDQueue heads, otherwise change (c);

(c) detect the step that rename is operated: in set Δ F, search r _s, if r _s∈ Δ F then produces rename operation script MV (r _s, r _t) be inserted into Q _MVQueue heads, otherwise change (d);

(d) step of processing replicate run: generate replicate run script CP (r _s, r _t) be inserted into Q _CPQueue heads.

3. the step of execute file rename playback script: from tail to a head traversal formation Q _CP, carry out each replicate run; Travel through formation Q from tail to head _MV, carry out each rename operation; Travel through formation Q from tail to head _RD, carry out the latter half that each rename is operated.

2) step of playback file content difference: to FS _cIn all differences record, utilize the playback algorithm of rsync, updating file;

3) playback client is created the step of new file operation: to FS _nIn all generate the record of new file, according to the record in new filename and file content, produce new file;

4) step of playback client delete file operation: scanning document FS _dAll records, calculate the pathdepth of each bar record, and, according to ordering successively, delete the file that every record is indicated according to the pathdepth descending sort.

Claims (1)

1. a remote file rapid synchronization method is characterized in that may further comprise the steps: the step of the file that the search file content is identical;

The step of spanned file state difference collection; The step of storage of file status difference collection and transmission; The step that file status difference collection is reset; Wherein:

(1) step of the file that the search file content is identical may further comprise the steps:

(1.1) step of comparison document length, whether the file size of a certain file equates in comparison document and the file set, if do not wait, changes (1.4), otherwise changes (1.2);

(1.2) step of comparison document hashed value, relatively the hashed value of two files in (1.1) if do not wait, is changeed (1.4), otherwise changes (1.3);

(1.3) filename that will have a same file content deposits the step of result set in;

(1.4) judge the step that traversal finishes, whether All Files travels through and finishes in the inspection file set, if traversal is finished, then withdraws from, otherwise changes (1.1);

(2) step of spanned file state difference collection may further comprise the steps:

(2.1) to belonging to t ₂All Files and t in the moment file set ₁Constantly file set, wherein t ₁＜t ₂, the step of the file that execution search file content is identical, Search Results deposits result set in, if result set is not empty, then changes (2.2), otherwise changes (2.3);

(2.2) detect the step that the file rename is operated, judge that filename is whether in the result set that (2.1) produce, if, file status no change then, otherwise file produces by duplicating, moving, represent file status difference by source filename and target part name, and add file status difference to file status difference and concentrate;

(2.3) step of the new file of detection is judged file f _iWhether belong to t ₁Moment file set, if, change (2.4), otherwise file f _iBe new file, file status difference is this document f _i, and add file status difference to file status difference and concentrate;

(2.4) step of calculation document content difference, by classical rsync algorithm computation file content difference, file status difference equals file content difference, adds file status difference to file status difference and concentrates;

(3) step of storage of file status difference collection and transmission may further comprise the steps:

(3.1) step that stores classifiedly of file status difference collection;

(3.2) open the step of the file of depositing file status difference;

(3.3) step of transmission file status difference sends to remote server with the content that opens file in (3.2) by network;

(3.4) check that file status difference sends the step that finishes, check whether all the difference storage file has sent to finish, finish, then withdraw from, otherwise switch to next difference storage file, change (3.2) if send;

(4) step of file status difference collection playback may further comprise the steps:

(4.1) step of playback client file rename operation comprises:

1. the step of all records of preprocessed file rename is calculated in the set of file destination name the pathdepth of file destination name in all records, and according to the pathdepth descending sort;

2. the step of spanned file rename playback script comprises:

(a) step of statement dique, statement dique Q _CP, Q _MVAnd Q _RD, Q wherein _CPPreserve replicate run, Q _MVPreserve the operation of rename operation and filename exchange first half, Q _RDPreserve the operation of execute file name exchange latter half;

(b) step of detection filename swap operation;

(c) detect the step of rename operation, concentrate search source file destination name,, then produce rename operation script and be inserted into the rename operation and filename exchange first half operation Q if this source file destination name belongs to the file difference collection at file difference _MVQueue heads, otherwise change (d);

(d) step of processing replicate run generates the replicate run script and is inserted into replicate run Q _CPQueue heads;

3. the step of execute file rename playback script travels through replicate run formation Q from tail to head _CP, carry out each replicate run; Travel through rename operation and filename exchange first half operation queue Q from tail to head _MV, carry out each rename operation; Exchange latter half operation queue Q from tail to head traversal filename _RD, carry out the latter half that each rename is operated;

(4.2) step of playback file content difference to the filename of generation file content difference and all differences record in the difference content file, is utilized the playback algorithm of rsync, updating file;

(4.3) playback client is created the step of new file operation, and the filename of newly-increased file and all in the file content file are generated the record of new file, and new filename and file content according in the record produce new file;

(4.4) step of playback client delete file operation, all records in the scanning deleted file calculate the pathdepth of every record, and according to the pathdepth descending sort, according to ordering successively, delete the file of every record indication.

Images