CN104079623B - Multistage cloud storage synchronisation control means and system - Google Patents
Multistage cloud storage synchronisation control means and system Download PDFInfo
- Publication number
- CN104079623B CN104079623B CN201410193621.6A CN201410193621A CN104079623B CN 104079623 B CN104079623 B CN 104079623B CN 201410193621 A CN201410193621 A CN 201410193621A CN 104079623 B CN104079623 B CN 104079623B
- Authority
- CN
- China
- Prior art keywords
- node
- file
- requesting node
- requesting
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of multistage cloud storage synchronisation control means and system, the multistage cloud storage synchronisation control means is included when receiving the synchronization request that multiple requesting nodes are sent, multiple requesting nodes are classified according to the type of node, and are ranked up sorted requesting node according to default priority orders；Data integrity detection is carried out to the requesting node according to the order of priority from high to low, data syn-chronization is carried out according to the incomplete requesting node of data integrity testing result control data and the complete requesting node of data；Data difference detection is carried out to the requesting node after synchronization according to requested node, when the data of the requesting node and requested node have differences, controls the requesting node to be synchronized with requested node.The present invention, which realizes, solves the problems, such as that synchronization request is excessive and causes network congestion in the same period, it is possible to increase the synchronous efficiency of multistage cloud storage.
Description
Technical field
The present invention relates to cloud storage field, more particularly to a kind of multistage cloud storage synchronisation control means and system.
Background technology
With the continuous innovation of computer technology, cloud storage technology has obtained fast development.Cloud storage is general in cloud computing
The new concept that extends in thought and developed, refers to by cluster application, grid or distributed file system
Etc. function, a large amount of various types of storage devices in network are gathered into collaborative work by application software, it is common right
The outer system that data storage and Operational Visit function are provided.Cloud storage technology can realize multiple main frames, front end storage clothes
Data syn-chronization between the business node such as device and rear end storage server, so that the data on each main frame and server are kept
It is newest, most complete.But when in a certain cloud storage system, if to initiate data simultaneously same for its multiple node within a period of time
During step request, it is easy for causing network congestion.
The above is only used for auxiliary and understands technical scheme, does not represent and recognizes that the above is existing skill
Art.
The content of the invention
It is a primary object of the present invention to provide a kind of multistage cloud storage synchronisation control means, it is intended to solve the same period
The problem of interior synchronization request excessively causes network congestion.
To achieve the above object, multistage cloud storage synchronisation control means provided by the invention, methods described include：
When receiving the synchronization request that multiple requesting nodes are sent, according to the type of node by multiple requesting nodes
Classified, and be ranked up sorted requesting node according to default priority orders；
Data integrity detection is carried out to the requesting node according to the order of priority from high to low, it is complete according to data
Property the complete requesting node of the incomplete requesting node of testing result control data and data carry out data syn-chronization；
According to requested node to after synchronization the requesting node carry out data difference detection, when the requesting node with
When the data of requested node have differences, the requesting node is controlled to be synchronized with requested node.
Preferably, it is described when receiving the synchronization request that multiple requesting nodes are sent, will be multiple according to the type of node
The requesting node is classified, and is wrapped the step of according to default priority orders, sorted requesting node is ranked up
Include：
When receiving the synchronization request that multiple requesting nodes are sent, the IP address of the requesting node is read；
The type of the requesting node is confirmed according to the IP address, and is entered the requesting node according to the type of node
Row classification；
Sorted requesting node is ranked up according to default priority orders；The type of the node includes service
Device and main frame, the priority of the server are higher than the main frame.
Preferably, it is described that data integrity detection is carried out to the requesting node according to the order of priority from high to low
Step includes：
According to the order of priority from high to low, the requesting node is controlled according to its internal All Files generation MD5 school
Test code；
Judged according to the MD5 check codes with the presence or absence of damage file in the requesting node, when in the requesting node
During in the presence of damage file, judge that the data of the requesting node are imperfect.
Preferably, the basis is requested the step of node carries out data difference detection to the requesting node after synchronization
Including：
Contrast the metadata or index of the requesting node and the file of requested node；
According to comparing result confirm the requesting node and/or requested node file whether be increase newly, delete or
Modification, if so, then judging that the data of the requesting node and requested node have differences.
Preferably, it is described when the data of requesting node and requested node have differences, to the requesting node and quilt
The step of requesting node synchronizes includes：
When the file for it is newly-increased when, then by the newly-increased file be added in the absence of this document requesting node or by
In requesting node；
When the file is deletes, then the file deleted in the requesting node or requested node is corresponded to；
When the file is changes, then according to the sequencing of modification time, will be updated in the file of rear modification
The file first changed.
Further, to achieve the above object, the present invention provides a kind of multistage cloud storage synchronous control system, the system
Including：
Processing module, for when receiving the synchronization request that multiple requesting nodes are sent, the type according to node will be more
The individual requesting node is classified, and is ranked up sorted requesting node according to default priority orders；
First detection module, for carrying out data integrity to the requesting node according to the order of priority from high to low
Detection；
First synchronization module, for according to the incomplete requesting node of data integrity testing result control data and data
Complete requesting node carries out data syn-chronization；
Second detection module, for carrying out data difference inspection to the requesting node after synchronization according to requested node
Survey；
Second synchronization module, for when the data of the requesting node and requested node have differences, described in control
Requesting node synchronizes with requested node.
Preferably, the processing module includes：
Reading unit, for when receiving the synchronization request that multiple requesting nodes are sent, reading the requesting node
IP address；
Taxon, for confirming the type of the requesting node according to the IP address, and will according to the type of node
The requesting node is classified；
Sequencing unit, for being ranked up sorted requesting node according to default priority orders；The node
Type include server and main frame, the priority of the server is higher than the main frame.
Preferably, the first detection module includes：
Control unit, according to the order of priority from high to low, the requesting node is controlled according to its internal All Files
Generate MD5 check codes；
First judging unit, for being judged to whether there is damage file in the requesting node according to the MD5 check codes,
When damage file in the requesting node be present, judge that the data of the requesting node are imperfect.
Preferably, second detection module includes：
Comparing unit, for contrasting the metadata or index of the requesting node and the file of requested node；
Second judging unit, for being according to the file of the comparing result confirmation requesting node and/or requested node
No is newly-increased, deletion or modification, if so, then judging that the data of the requesting node and requested node have differences.
Preferably, second synchronization control module includes：
First synchronization unit, it is somebody's turn to do for when the file is newly-increased, then being added to the newly-increased file to be not present
In the requesting node of file or requested node；
Second synchronization unit, for when the file is deletes, then corresponding to and deleting the requesting node or requested section
The file in point；
3rd synchronization unit, for that when the file is changes, then according to the sequencing of modification time, will be repaiied after
The file changed is updated to the file formerly changed.
The present invention is by the way that multiple nodes that synchronization request is initiated within the same period are classified, and according to default
Sorted requesting node is ranked up by priority orders, and then the request is saved according to the order of priority from high to low
Row data integrity detection is clicked through, and it is complete according to the incomplete requesting node of integrity detection output control data and data
Requesting node carry out data syn-chronization, after the data of requesting node are complete, continue to carry out it data difference detection, and in institute
When stating the data of requesting node and requested node and having differences, the requesting node is controlled to be synchronized with requested node,
It is achieved thereby that solve the problems, such as that synchronization request is excessive and causes network congestion in the same period.
Brief description of the drawings
Fig. 1 is the schematic flow sheet of the multistage embodiment of cloud storage synchronisation control means one of the present invention；
Fig. 2 is an embodiment schematic flow sheet of step S10 in Fig. 1；
Fig. 3 is an embodiment schematic flow sheet of step S20 in Fig. 1；
Fig. 4 is an embodiment schematic flow sheet of step S30 in Fig. 1；
Fig. 5 is an embodiment schematic flow sheet of step S40 in Fig. 1；
Fig. 6 is the priority scheduling schematic diagram using each node after the multistage cloud storage synchronisation control means of the present invention；
Fig. 7 is the structural representation of the multistage embodiment of cloud storage synchronous control system one of the present invention；
Fig. 8 is the structural representation of the embodiment of processing module one in Fig. 7；
Fig. 9 is the structural representation of the embodiment of first detection module one in Fig. 7；
Figure 10 is the structural representation of the embodiment of the second detection module one in Fig. 7；
Figure 11 is the structural representation of the embodiment of the second synchronization module one in Fig. 7.
The realization, functional characteristics and advantage of the object of the invention will be described further referring to the drawings in conjunction with the embodiments.
Embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
In the present invention, node refers to that a computer or other equipment have independent address and with transmission or reception with one
The network of data function is connected.Node can be work station, client, the network user or personal computer, can also be server,
The equipment of printer and other network connections, i.e. each work station, server, terminal device, the network equipment and possess oneself
The equipment of own unique network address is all network node, and each network node can obtain unique IP address by network interface card.
Reference picture 1, Fig. 1 are the schematic flow sheet of the multistage embodiment of cloud storage synchronisation control means one of the present invention；Such as Fig. 1 institutes
Show, the present embodiment multistage cloud storage synchronisation control means comprises the following steps：
Step S10, when receiving the synchronization request that multiple requesting nodes are sent, according to the type of node will be multiple
Requesting node is classified, and is ranked up sorted requesting node according to default priority orders；
Each node has its corresponding type, and the type of node can be obtained by the type designation of recognition node,
The node type is divided into server and main frame in the present embodiment.When node is server, then server category is divided to
Type, when node is main frame, then it is divided to Host Type.Wherein, priority orders information is prestored in each node so that
After each node to initiating request is classified, the priority orders of each node by type can be ranked up, and sorted
When, the high type of priority level is come before, behind the low type of priority level comes, for example, predetermined server is preferential
Level is when being higher than main frame, then will be that the requesting node of type of server comes and is before the requesting node of Host Type, i.e. server
In requesting node prior to the requesting node in main frame.By the type of recognition node, then divided according to node type
Class processing, to realize that each requesting node is handled successively in batches, reduce synchronization request and excessively take Internet resources.
Step S20, data integrity detection is carried out to the requesting node according to the order of priority from high to low；
The data of storage in node may be damaged or misplace, therefore, it is necessary to storing number in node
According to being repaired, in this step, according to the priority orders sequenced, it is complete that data are carried out successively to the preceding requesting node of order
Whole property detection.During detection, some file of identical in two nodes can be contrasted, confirms that it is wrong with the presence or absence of defect or byte
Position etc., to confirm that the data in the node are imperfect.
Step S30, completely please according to the incomplete requesting node of data integrity testing result control data and data
Node is asked to carry out data syn-chronization；
The present embodiment is specifically by the incomplete requesting node of data in same class row and the complete requesting node of data
Same file carry out data syn-chronization, to ensure that the data of each requesting node in this type are identical.It is understood that work as
Node type is server and main frame, and server priority then first realizes that each requesting node in server is first complete when main frame
Into synchronization, then realizing each requesting node in main frame complete it is synchronous, finally realize it is all types of in requesting node data
Integrality.
It is understood that to prevent the data in requesting node from once incomplete situation also be present afterwards synchronous,
It in example is further carried out, can be arranged on after execution of step S30, return and perform the step S20.That is, to synchronization
The requesting node afterwards carries out integrity detection again, to ensure the complete, reliable of data.
Step S40, data difference detection is carried out to the requesting node after synchronization according to requested node；
In this step, the file in the requesting node after synchronization is compared with requested node, judges to ask
Data whether there is difference in node and requested node, predominantly detect the file in requesting node and/or requested node
With the presence or absence of newly-increased, deletion or modification.Can be that requesting node is examined using requested node as reference in the present embodiment
Survey, or requested node is detected by reference of requesting node, or be cross-referenced detection between the two, i.e.
Both the file in confirmation request node was with the presence or absence of newly-increased, deletion or modification, and whether the file in the requested node of confirmation again
In the presence of newly-increased, deletion or modification, so ensure that the data in requested node and requesting node are all kept after synchronization
It is newest, most complete.
Step S50, when the data of the requesting node and requested node have differences, control the requesting node with
Requested node synchronizes.
It should be appreciated that in this step, requesting node with requested node can be one-way synchronization or two-way
It is synchronous, wherein, one-way synchronization is included the data syn-chronization on requesting node to requested node, and by the number on requested node
According to being synchronized on requesting node；Bi-directional synchronization is that control data node carries out data syn-chronization between each other with requested node, is made
Data in both all keep newest, most complete.
To sum up, the inventive method by by within the same period initiate synchronization request multiple nodes classified, and
Sorted requesting node is ranked up according to default priority orders, then the order according to priority from high to low is right
The requesting node carries out data integrity detection, and according to the incomplete requesting node of integrity detection output control data
Requesting node complete with data carries out data syn-chronization, after the data of requesting node are complete, continues to carry out data difference to it
Detection, and when the data of the requesting node and requested node have differences, control the requesting node and requested section
Point synchronizes, it is achieved thereby that solving the problems, such as that synchronization request is excessive and causes network congestion in the same period, and protects
It is accurate after synchronization each node data is demonstrate,proved, completely.
Further, reference picture 2, Fig. 2 are an embodiment schematic flow sheet of step S10 in Fig. 1；
As shown in Fig. 2 wherein step S10 specifically includes following steps：
Step S101, when receiving the synchronization request that multiple requesting nodes are sent, with reading the IP of the requesting node
Location；
Step S102, the type of the requesting node is confirmed according to the IP address, and according to the type of node by described in
Requesting node is classified；
Step S103, sorted requesting node is ranked up according to default priority orders；
Each node has unique IP address, and the type of the node, this implementation can be obtained according to its IP address
In example, the type of the node includes server and main frame, and the priority of the server is higher than the main frame.Wherein, it is a certain
When type includes multiple nodes, each node can be ranked up according to network connection speed, to the faster section of network connection speed
Point priority treatment.
Further, reference picture 3, Fig. 3 are an embodiment schematic flow sheet of step S20 in Fig. 1；
As shown in figure 3, wherein step S20 specifically includes following steps：
Step S201, the order according to priority from high to low, the requesting node is controlled according to its internal All Files
Generate MD5 check codes；
Step S202, judge to ask when described with the presence or absence of damage file in the requesting node according to the MD5 check codes
When asking in node in the presence of damage file, judge that the data of the requesting node are imperfect；
MD5 is that a kind of irreversible character string becomes scaling method, is given birth to by control data node according to its internal All Files
It is known that whether the existing file in the node is consistent with original into corresponding MD5 check codes, so as to determine described ask
Ask the data of node whether complete.
Further, reference picture 4, Fig. 4 are an embodiment schematic flow sheet of step S30 in Fig. 1；
As shown in figure 4, wherein step S30 specifically includes following steps：
Step S301, the metadata or index of the requesting node and the file of requested node are contrasted；
Step S302, according to comparing result confirm the requesting node and/or requested node file whether be increase newly,
Delete or change, if so, then judging that the data of the requesting node and requested node have differences.
In the present embodiment, requesting node and identical file in requested node are contrasted, if in requested node
File it is more compared to the file number in requesting node, or file in requesting node is compared to the text in requested node
Part number is more, then it is newly-increased to judge the file；If the file in requested node is compared to the file number in requesting node
Less, or the file in requesting node is few compared to the file number in requested node, then judges the file to delete；If
File in requested node is consistent with the file number in requesting node, and file and/or request section in requested node
When file in point has modification, then judge the file for modification.
Further, reference picture 5, Fig. 5 are an embodiment schematic flow sheet of step S40 in Fig. 1；
As shown in figure 5, wherein step 40 specifically includes following steps：
Step S401, when the file is newly-increased, then the newly-increased file is added to asking in the absence of this document
Ask in node or requested node；
Step S402, when the file is deletes, then the institute deleted in the requesting node or requested node is corresponded to
State file；
Step S403, when the file is changes, then according to the sequencing of modification time, by the file of rear modification
It is updated to the file formerly changed.
In the present embodiment, when detecting the file in requested node relative to the file of requesting node to increase newly, i.e.,
File number in requested node is more than original, and now, additional file will be updated to requesting node, that is, download；When
When detecting the file in requesting node relative to the file of requested node to increase newly, i.e., the file number ratio in requesting node
Originally more, now, additional file will be updated to requested node, that is, upload.
When detecting the file in requested node relative to the file of requesting node to reduce, i.e., in requested node
File number it is fewer than original, now, being requested the file of reduction in node will delete from requesting node.Wherein, request section
Point, which receives, needs the file deleted, and searches corresponding file in requested node, if file to be deleted is present, deletes pair
The index of file, file label and the file answered.When detecting text of the file in requesting node relative to requested node
When part is reduces, i.e., file number in requesting node is fewer than original, and now, the file of reduction will be from being requested in requesting node
Deleted in node.Wherein, it is requested node and receives the file for needing to delete, and corresponding file is searched in requesting node, such as
Fruit file to be deleted is present, then the index of file, file label and file corresponding to deletion.
Relative to some all existing file in requesting node and requested node, it is assumed that be W, if detecting request section
When the file W of point and requested node has modification, then compare what requesting node was changed with file W in requested node
The sequencing of time, if the file W in requesting node is first relative to the file W modification times in requested node, and by
File W in requesting node relative to the file W modification times in requesting node rear, then by the file W in requested node
The file W being updated in requesting node, make the file W in requesting node consistent with the file W in requested node.
It should be noted that when data syn-chronization is carried out between requesting node and requested node, at the both ends of synchronization point
Fixed size M block is not divided documents into, and calculates each piece of check code.Wherein, the newer one end of file data will
Check code is sent to the older one end of file data, i.e., end to be synchronized, end to be synchronized is by searching for compared with all sizes in ancient deed
Position compared with the data block that identical check code is carried in new file for M block, then sent more to the newer one end of file data
Newly compared with the request of ancient deed, and start synchronization.
Reference picture 6, Fig. 6 are shown using the priority scheduling of each requesting node after the multistage cloud storage synchronisation control means of the present invention
It is intended to.
Such as Fig. 6, sent as multiple IP address unique host node R1, R2 ... Rn-1, Rn, while to requested node S1
During synchronization request, it is assumed that (IP address m) R1 obtain higher priority to a certain requesting node, and enter at buffering queue wait
Reason.Now, the IP address information in the solicited message message that node S1 is sent by analysis request node is requested to search simultaneously
The type of requesting node is determined, and is ranked up each requesting node according to default priority orders.
After sequence, each requesting node will carry out a completeness check to all files that presently, there are of its inside.Its
In, requesting node that completeness check passes through, then carry out data difference inspection.
Data difference, which is examined, finds existing file B, C, D in requesting node R1, and file A is added before synchronization；Quilt
Existing file B and file C in requesting node S1, and reduce file D before synchronization, now, while it is present in requesting node
File B in R1 and requested node S1 does not have found data difference, but detects data difference in file C be present, and asks section
Point R1 is compared to requested node S1 transmitting file C on rear.
Newly-increased synchronization policy is taken for file A in requesting node R1, i.e., is uploaded to file A in requested node S1
Deletion synchronization policy is taken for file D in requesting node R1, i.e., is deleted file D from requested node R 1.
For file B, because data difference is not present in the file B in requesting node R1 and requested node R 1, so together
This file is crossed during step.
For file C, because the C files (CR1) on requesting node R1 are compared to the C files on requested node S1
(CS1) in rear modification, due to file CR1 and file CS1 similar documents each other, therefore S1 is arrived in its renewal by the C files (CR1) on R1
On, its process is as follows：
File CR1 is divided into equal in magnitude or not etc. data block at requesting node R1 ends；
And at requesting node R1 ends, correspond to each data block and generate a check code, and record data block length and should
Data block is in the offset in file B.
Then, requesting node R1 sends these data block informations to requested node S1；Requested node S1 ends use
File CS1 is divided into equal in magnitude or not etc. data block by same data block mode, and sent with requesting node R1
Data message is scanned for matching, and generates difference coding information, and difference coding information is sent to request section by requested node S1
Point R1, and reconstruct file CS1 instruction is sent simultaneously.So far, synchronously complete.
Accordingly, the present invention also provides a kind of multistage cloud storage synchronous control system, reference picture 7, and Fig. 7 is present invention multistage
The structural representation of the embodiment of cloud storage synchronous control system one；As shown in fig. 7, the system is examined including processing module 100, first
Survey module 200, the first synchronization module 300 and the second detection module 400 and the second synchronization module 500；
Wherein, processing module 100, for when receiving the synchronization request that multiple requesting nodes are sent, according to node
Type is classified multiple requesting nodes, and is arranged sorted requesting node according to default priority orders
Sequence；
Each node has its corresponding type, and the type of node can be obtained by the type designation of recognition node,
The node type is divided into server and main frame in the present embodiment.When node is server, then server category is divided to
Type, when node is main frame, then it is divided to Host Type.Wherein, priority orders information is prestored in each node so that
After each node to initiating request is classified, the priority orders of each node by type can be ranked up, and sorted
When, the high type of priority level is come before, behind the low type of priority level comes, for example, predetermined server is preferential
Level is when being higher than main frame, then will be that the requesting node of type of server comes and is before the requesting node of Host Type, i.e. server
In requesting node prior to the requesting node in main frame.By the type of recognition node, then divided according to node type
Class processing, to realize that each requesting node is handled successively in batches, reduce synchronization request and excessively take Internet resources.
First detection module 200, it is complete for carrying out data to the requesting node according to the order of priority from high to low
Whole property detection；The data of storage in node may be damaged or misplace, therefore, it is necessary to storing number in node
According to being repaired, wherein, according to the priority orders sequenced, data integrity is carried out successively to the preceding requesting node of order
Detection.During detection, some file of identical in two nodes can be contrasted, confirms that it whether there is defect or erroneous bytes
Deng to confirm that the data in the node are imperfect.
First synchronization module 300, for according to the incomplete requesting node of data integrity testing result control data with
The complete requesting node of data carries out data syn-chronization；The present embodiment is specifically by the incomplete requesting node of data in same class row
Data syn-chronization is carried out with the same file in the complete requesting node of data, to ensure the number of each requesting node in this type
According to identical.It is understood that when node type is server and main frame, and server priority then first realizes clothes when main frame
Each requesting node in business device first completes synchronization, is then realizing that the completion of each requesting node in main frame is synchronous, final realization is each
The integrality of the data of requesting node in type.
Second detection module 400, for carrying out data difference to the requesting node after synchronization according to requested node
Detection；File in the requesting node after synchronization is compared with requested node, requesting node is judged and is asked
Ask data in node to whether there is difference, predominantly detect file in requesting node and/or requested node with the presence or absence of it is newly-increased,
Delete or change.Can be that requesting node is detected using requested node as reference in the present embodiment, or it is requested
Node is detected by reference of requesting node, or is cross-referenced detection between the two, i.e. both in confirmation request node
File with the presence or absence of it is newly-increased, delete or modification, confirm again file in requested node with the presence or absence of it is newly-increased, delete or
Modification, so ensures that the data in requested node and requesting node all keep newest, most complete after synchronization.
Second synchronization module 500, for when the data of the requesting node and requested node have differences, controlling institute
Requesting node is stated to synchronize with requested node.
It should be appreciated that requesting node and requested node can be one-way synchronization or bi-directional synchronization, its
In, one-way synchronization is included the data syn-chronization on requesting node to requested node, and by the data syn-chronization on requested node
To requesting node；Bi-directional synchronization is that control data node carries out data syn-chronization between each other with requested node, is made in both
Data all keep newest, most complete.
It is understood that to prevent the data in requesting node from once incomplete situation also be present afterwards synchronous,
In example is further carried out, the first detection module 200 is additionally operable to incomplete in the control data of the first synchronization module 300
After requesting node carries out data syn-chronization with the complete requesting node of data, continue to carry out the requesting node after synchronization again
Data integrity detects, to ensure the complete, reliable of data.
To sum up, apparatus of the present invention will initiate multiple nodes of synchronization request by processing module 100 within the same period
Classified, and be ranked up sorted requesting node according to default priority orders, then pass through the first detection mould
Block 200 carries out data integrity detection according to the order of priority from high to low to the requesting node, by first after having detected
Synchronization module 300 is carried out according to the incomplete requesting node of integrity detection output control data and the complete requesting node of data
Data syn-chronization, continue to carry out data difference inspection to it after the data of requesting node are complete, then by the second detection module 400
Survey, and when the data of the requesting node and requested node have differences, controlled by second synchronization module 500 described
Requesting node synchronizes with requested node, it is achieved thereby that solving synchronization request in the same period excessively causes network
The problem of congestion, and ensure that each node data is accurate after synchronization, completely.
Further, reference picture 8, Fig. 8 are the structural representation of the embodiment of processing module one in Fig. 7；
As shown in figure 8, the processing module 100 includes：
Reading unit 101, for when receiving the synchronization request that multiple requesting nodes are sent, reading the requesting node
IP address；
Taxon 102, for confirming the type of the requesting node according to the IP address, and according to the type of node
The requesting node is classified；
Sequencing unit 103, for being ranked up sorted requesting node according to default priority orders；The section
The type of point includes server and main frame, and the priority of the server is higher than the main frame.
Each node has unique IP address, and the type of the node, this implementation can be obtained according to its IP address
In example, the type of the node includes server and main frame, and the priority of the server is higher than the main frame.Wherein, it is a certain
When type includes multiple nodes, each node can be ranked up according to network connection speed, to the faster section of network connection speed
Point priority treatment.
Further, reference picture 9, Fig. 9 are the structural representation of the embodiment of first detection module 200 1 in Fig. 7；
As shown in figure 8, the first detection module 200 includes：
Control unit 201, according to the order of priority from high to low, the requesting node is controlled according to its internal all text
Part generates MD5 check codes；
First judging unit 202, for being judged according to the MD5 check codes in the requesting node with the presence or absence of damage text
Part, when damage file in the requesting node be present, judge that the data of the requesting node are imperfect.
MD5 is that a kind of irreversible character string becomes scaling method, is given birth to by control data node according to its internal All Files
It is known that whether the existing file in the node is consistent with original into corresponding MD5 check codes, so as to determine described ask
Ask the data of node whether complete.
Further, reference picture 10, Figure 10 are the structural representation of the embodiment of the second detection module one in Fig. 7；
As shown in Figure 10, second detection module 400 includes：
Comparing unit 401, for contrasting the metadata or index of the requesting node and the file of requested node；
Second judging unit 402, for confirming the file of the requesting node and/or requested node according to comparing result
Whether it is to increase newly, delete or change, if so, then judging that the data of the requesting node and requested node have differences.
In the present embodiment, requesting node and identical file in requested node are contrasted, if in requested node
File it is more compared to the file number in requesting node, or file in requesting node is compared to the text in requested node
Part number is more, then it is newly-increased to judge the file；If the file in requested node is compared to the file number in requesting node
Less, or the file in requesting node is few compared to the file number in requested node, then judges the file to delete；If
File in requested node is consistent with the file number in requesting node, and file and/or request section in requested node
When file in point has modification, then judge the file for modification.
Further, reference picture 11, Figure 11 are the structural representation of the embodiment of the second synchronization module one in Fig. 7；
As shown in Figure 10, the second synchronization module 500 includes described in the present embodiment：
First synchronization unit 501, for when the file is increases newly, being then added to and being not present by the newly-increased file
In the requesting node of this document or requested node；
Second synchronization unit 502, for when the file is deletes, then corresponding to and deleting the requesting node or requested
The file in node；
3rd synchronization unit 503, for when the file for modification when, then, will be according to the sequencing of modification time
The file changed afterwards is updated to the file formerly changed.
In the present embodiment, when detecting the file in requested node relative to the file of requesting node to increase newly, i.e.,
File number in requested node is more than original, and now, additional file will be updated to requesting node, that is, download；When
When detecting the file in requesting node relative to the file of requested node to increase newly, i.e., the file number ratio in requesting node
Originally more, now, additional file will be updated to requested node, that is, upload.
When detecting the file in requested node relative to the file of requesting node to reduce, i.e., in requested node
File number it is fewer than original, now, being requested the file of reduction in node will delete from requesting node.Wherein, request section
Point, which receives, needs the file deleted, and searches corresponding file in requested node, if file to be deleted is present, deletes pair
The index of file, file label and the file answered.When detecting text of the file in requesting node relative to requested node
When part is reduces, i.e., file number in requesting node is fewer than original, and now, the file of reduction will be from being requested in requesting node
Deleted in node.Wherein, it is requested node and receives the file for needing to delete, and corresponding file is searched in requesting node, such as
Fruit file to be deleted is present, then the index of file, file label and file corresponding to deletion.
Relative to some all existing file in requesting node and requested node, it is assumed that be W, if detecting request section
When the file W of point and requested node has modification, then compare what requesting node was changed with file W in requested node
The sequencing of time, if the file W in requesting node is first relative to the file W modification times in requested node, and by
File W in requesting node relative to the file W modification times in requesting node rear, then by the file W in requested node
The file W being updated in requesting node, make the file W in requesting node consistent with the file W in requested node.
It should be noted that when data syn-chronization is carried out between requesting node and requested node, at the both ends of synchronization point
Fixed size M block is not divided documents into, and calculates each piece of check code.Wherein, the newer one end of file data will
Check code is sent to the older one end of file data, i.e., end to be synchronized, end to be synchronized is by searching for compared with all sizes in ancient deed
Position compared with the data block that identical check code is carried in new file for M block, then sent more to the newer one end of file data
Newly compared with the request of ancient deed, and start synchronization.
Reference picture 6, Fig. 6 are shown using the priority scheduling of each requesting node after the multistage cloud storage synchronous control system of the present invention
It is intended to.
Such as Fig. 6, sent as multiple IP address unique host node R1, R2 ... Rn-1, Rn, while to requested node S1
During synchronization request, it is assumed that (IP address m) R1 obtain higher priority to a certain requesting node, and enter at buffering queue wait
Reason.Now, the IP address information in the solicited message message that node S1 is sent by analysis request node is requested to search simultaneously
The type of requesting node is determined, and is ranked up each requesting node according to default priority orders.
After sequence, each requesting node will carry out a completeness check to all files that presently, there are of its inside.Its
In, requesting node that completeness check passes through, then carry out data difference inspection.
Data difference, which is examined, finds existing file B, C, D in requesting node R1, and file A is added before synchronization；Quilt
Existing file B and file C in requesting node S1, and reduce file D before synchronization, now, while it is present in requesting node
File B in R1 and requested node S1 does not have found data difference, but detects data difference in file C be present, and asks section
Point R1 is compared to requested node S1 transmitting file C on rear.
Newly-increased synchronization policy is taken for file A in requesting node R1, i.e., is uploaded to file A in requested node S1
Deletion synchronization policy is taken for file D in requesting node R1, i.e., is deleted file D from requested node R 1.
For file B, because data difference is not present in the file B in requesting node R1 and requested node R 1, so together
This file is crossed during step.
For file C, because the C files (CR1) on requesting node R1 are compared to the C files on requested node S1
(CS1) in rear modification, due to file CR1 and file CS1 similar documents each other, therefore S1 is arrived in its renewal by the C files (CR1) on R1
On, its process is as follows：
File CR1 is divided into equal in magnitude or not etc. data block at requesting node R1 ends；
And at requesting node R1 ends, correspond to each data block and generate a check code, and record data block length and should
Data block is in the offset in file B.
Then, requesting node R1 sends these data block informations to requested node S1；Requested node S1 ends use
File CS1 is divided into equal in magnitude or not etc. data block by same data block mode, and sent with requesting node R1
Data message is scanned for matching, and generates difference coding information, and difference coding information is sent to request section by requested node S1
Point R1, and reconstruct file CS1 instruction is sent simultaneously.So far, synchronously complete.
The embodiments of the present invention are for illustration only, do not represent the quality of embodiment.Embodiment party more than
The description of formula, it is required general that those skilled in the art can be understood that above-described embodiment method can add by software
The mode of hardware platform is realized, naturally it is also possible to which by hardware, but the former is more preferably embodiment in many cases.It is based on
Such understanding, the part that technical scheme substantially contributes to prior art in other words can be with software products
Form embody, the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disc, CD), including
Some instructions are causing a station terminal equipment (can be mobile phone, computer, server, or network equipment etc.) to perform this hair
Method described in bright each embodiment.
The preferred embodiments of the present invention are these are only, are not intended to limit the scope of the invention, it is every to utilize this hair
The equivalent structure or equivalent flow conversion that bright specification and accompanying drawing content are made, or directly or indirectly it is used in other related skills
Art field, is included within the scope of the present invention.
Claims (8)
- A kind of 1. multistage cloud storage synchronisation control means, it is characterised in that including：When receiving the synchronization request that multiple requesting nodes are sent, multiple requesting nodes are carried out according to the type of node Classification, and be ranked up sorted requesting node according to default priority orders；Data integrity detection is carried out to the requesting node according to the order of priority from high to low, examined according to data integrity Survey the incomplete requesting node of output control data and carry out data syn-chronization with the complete requesting node of data；Data difference detection is carried out to the requesting node after synchronization according to requested node, when the requesting node and asked When asking the data of node to have differences, the requesting node is controlled to be synchronized with requested node；Wherein, described the step of carrying out data integrity detection to the requesting node according to the order of priority from high to low, wraps Include：According to the order of priority from high to low, the requesting node is controlled according to its internal All Files generation MD5 check code；Judge exist in the requesting node with the presence or absence of damage file when in the requesting node according to the MD5 check codes When damaging file, judge that the data of the requesting node are imperfect.
- 2. multistage cloud storage synchronisation control means as claimed in claim 1, it is characterised in that described to receive multiple requests During the synchronization request that node is sent, multiple requesting nodes are classified according to the type of node, and according to default excellent The step of sorted requesting node is ranked up by first level order includes：When receiving the synchronization request that multiple requesting nodes are sent, the IP address of the requesting node is read；The type of the requesting node is confirmed according to the IP address, and is divided the requesting node according to the type of node Class；Sorted requesting node is ranked up according to default priority orders；The type of the node include server and Main frame, the priority of the server are higher than the main frame.
- 3. multistage cloud storage synchronisation control means as claimed in claim 1, it is characterised in that the basis is requested node pair The step of requesting node after synchronization carries out data difference detection includes：Contrast the metadata or index of the requesting node and the file of requested node；Whether the file that the requesting node and/or requested node are confirmed according to comparing result is to increase newly, delete or change, If so, then judge that the data of the requesting node and requested node have differences.
- 4. multistage cloud storage synchronisation control means as claimed in claim 3, it is characterised in that described when requesting node is with being asked When asking the data of node to have differences, the step of being synchronized to the requesting node with requested node, includes：When the file is increases newly, then the newly-increased file is added to the requesting node or requested in the absence of this document In node；When the file is deletes, then the file deleted in the requesting node or requested node is corresponded to；When the file is changes, then according to the sequencing of modification time, formerly repaiied being updated in the file of rear modification The file changed.
- A kind of 5. multistage cloud storage synchronous control system, it is characterised in that including：Processing module, for when receiving the synchronization request that multiple requesting nodes are sent, according to the type of node by multiple institutes State requesting node to be classified, and be ranked up sorted requesting node according to default priority orders；First detection module, for carrying out data integrity inspection to the requesting node according to the order of priority from high to low Survey；First synchronization module, for complete according to the incomplete requesting node of data integrity testing result control data and data Requesting node carry out data syn-chronization；Second detection module, for carrying out data difference detection to the requesting node after synchronization according to requested node；Second synchronization module, for when the data of the requesting node and requested node have differences, controlling the request Node synchronizes with requested node；Wherein, the first detection module includes：Control unit, according to the order of priority from high to low, the requesting node is controlled according to its internal All Files generation MD5 check codes；First judging unit, for being judged to work as institute with the presence or absence of damage file in the requesting node according to the MD5 check codes When stating in requesting node in the presence of damage file, judge that the data of the requesting node are imperfect.
- 6. multistage cloud storage synchronous control system as claimed in claim 5, it is characterised in that the processing module includes：Reading unit, for when receiving the synchronization request that multiple requesting nodes are sent, with reading the IP of the requesting node Location；Taxon, for confirming the type of the requesting node according to the IP address, and according to the type of node by described in Requesting node is classified；Sequencing unit, for being ranked up sorted requesting node according to default priority orders；The class of the node Type includes server and main frame, and the priority of the server is higher than the main frame.
- 7. multistage cloud storage synchronous control system as claimed in claim 6, it is characterised in that the second detection module bag Include：Comparing unit, for contrasting the metadata or index of the requesting node and the file of requested node；Second judging unit, for confirmed according to comparing result the requesting node and/or requested node file whether be Newly-increased, deletion or modification, if so, then judging that the data of the requesting node and requested node have differences.
- 8. multistage cloud storage synchronous control system as claimed in claim 7, it is characterised in that second synchronization control module Including：First synchronization unit, for when the file is increases newly, being then added to the newly-increased file and this document being not present Requesting node or requested node in；Second synchronization unit, for when the file is deletes, then corresponding to and deleting in the requesting node or requested node The file；3rd synchronization unit, for when the file for modification when, then according to the sequencing of modification time, by rear modification File is updated to the file formerly changed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410193621.6A CN104079623B (en) | 2014-05-08 | 2014-05-08 | Multistage cloud storage synchronisation control means and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410193621.6A CN104079623B (en) | 2014-05-08 | 2014-05-08 | Multistage cloud storage synchronisation control means and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104079623A CN104079623A (en) | 2014-10-01 |
| CN104079623B true CN104079623B (en) | 2018-03-20 |
Family
ID=51600663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410193621.6A Active CN104079623B (en) | 2014-05-08 | 2014-05-08 | Multistage cloud storage synchronisation control means and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104079623B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104378427A (en) * | 2014-11-14 | 2015-02-25 | 浪潮电子信息产业股份有限公司 | Method for keeping consistency of cluster key data |
| US9843501B2 (en) * | 2015-07-02 | 2017-12-12 | Qualcomm Incorporated | Systems and methods for incorporating devices into a medical data network |
| CN106254373B (en) * | 2016-08-31 | 2019-12-27 | 北京信安世纪科技股份有限公司 | Digital certificate synchronization method, digital signature server and digital certificate synchronization system |
| CN108156086B (en) * | 2017-12-19 | 2022-04-22 | 北京奇艺世纪科技有限公司 | Policy rule issuing method and device |
| CN110602168B (en) * | 2019-08-13 | 2022-03-01 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN111932405A (en) * | 2020-09-15 | 2020-11-13 | 天津中新智冠信息技术有限公司 | Method and system for determining power distribution state, electronic device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101826073A (en) * | 2009-03-06 | 2010-09-08 | 华为技术有限公司 | Synchronous method, apparatus and system for distributed database |
| CN102902746A (en) * | 2012-09-18 | 2013-01-30 | 杭州勒卡斯广告策划有限公司 | Method, device and system for processing mass data |
| CN102932621A (en) * | 2011-08-08 | 2013-02-13 | 杭州海康威视数字技术股份有限公司 | Method and device for storing data |
| CN103152398A (en) * | 2013-02-07 | 2013-06-12 | 中国科学院计算技术研究所 | Method and system for file synchronization among multiple terminal files |
| CN103180842A (en) * | 2010-10-28 | 2013-06-26 | Sk电信有限公司 | Cloud computing system and data synchronization method therefor |
| CN103327037A (en) * | 2012-03-20 | 2013-09-25 | 中兴通讯股份有限公司 | Data synchronizing method and device |
| CN103761162A (en) * | 2014-01-11 | 2014-04-30 | 深圳清华大学研究院 | Data backup method of distributed file system |
-
2014
- 2014-05-08 CN CN201410193621.6A patent/CN104079623B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101826073A (en) * | 2009-03-06 | 2010-09-08 | 华为技术有限公司 | Synchronous method, apparatus and system for distributed database |
| CN103180842A (en) * | 2010-10-28 | 2013-06-26 | Sk电信有限公司 | Cloud computing system and data synchronization method therefor |
| CN102932621A (en) * | 2011-08-08 | 2013-02-13 | 杭州海康威视数字技术股份有限公司 | Method and device for storing data |
| CN103327037A (en) * | 2012-03-20 | 2013-09-25 | 中兴通讯股份有限公司 | Data synchronizing method and device |
| CN102902746A (en) * | 2012-09-18 | 2013-01-30 | 杭州勒卡斯广告策划有限公司 | Method, device and system for processing mass data |
| CN103152398A (en) * | 2013-02-07 | 2013-06-12 | 中国科学院计算技术研究所 | Method and system for file synchronization among multiple terminal files |
| CN103761162A (en) * | 2014-01-11 | 2014-04-30 | 深圳清华大学研究院 | Data backup method of distributed file system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104079623A (en) | 2014-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104079623B (en) | Multistage cloud storage synchronisation control means and system | |
| CN106936441B (en) | Data compression method and device | |
| US10262038B2 (en) | Answerer extraction system, answerer extraction method, and non-transitory computer-readable medium | |
| WO2018177190A1 (en) | Method and device for synchronizing blockchain data | |
| CN107483627A (en) | A kind of file distributing, method for down loading, Distributor, client and system | |
| EP2985707A1 (en) | Method and apparatus for finding file in storage device and router and medium | |
| CN104348859B (en) | File synchronisation method, device, server, terminal and system | |
| CN105450785B (en) | File transmission method and device | |
| US20170109371A1 (en) | Method and Apparatus for Processing File in a Distributed System | |
| US20180107686A1 (en) | Search method and apparatus | |
| WO2016000507A1 (en) | Traffic-saving mode search service method, server, client and system | |
| CN110858172A (en) | Automatic test code generation method and device | |
| CN110019263B (en) | Information storage method and device | |
| US20170199889A1 (en) | Method and device for identifying junk picture files | |
| CN109815112B (en) | Data debugging method and device based on functional test and terminal equipment | |
| CN105989158A (en) | Screenshot distinguishing method and equipment | |
| CN113377653B (en) | Method and device for generating test cases | |
| CN112988916B (en) | Full and incremental synchronization method, apparatus and storage medium for Clickhouse | |
| CN110572422A (en) | Data downloading method and device | |
| CN104750741A (en) | Invalid link processing method and invalid link processing device | |
| CN103368817A (en) | Method and device for sending and receiving e-mail | |
| CN106302604B (en) | Data transmission method and device | |
| CN109254921A (en) | Application version verification method, device, computer equipment and storage medium | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN109254954A (en) | Document handling method and device calculate equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20141001 Assignee: Liu Yi Assignor: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. Contract record no.: 2014440020487 Denomination of invention: Method and system for controlling multilevel cloud storage synchrony License type: Common License Record date: 20141230 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model | ||
| EC01 | Cancellation of recordation of patent licensing contract |
Assignee: Liu Yi Assignor: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. Contract record no.: 2014440020487 Date of cancellation: 20161025 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190905 Address after: 100089 Floor 1-4, No. 2 Building, No. 9 Courtyard, Dijin Road, Haidian District, Beijing Patentee after: Beijing Toyou Feiji Electronics Co., Ltd. Address before: 518000 Room 1402, Feiyada Science and Technology Building, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. |
|
| TR01 | Transfer of patent right |