CN103716384A - Method and device for realizing cloud storage data synchronization in cross-data-center manner - Google Patents

Method and device for realizing cloud storage data synchronization in cross-data-center manner Download PDF

Info

Publication number
CN103716384A
CN103716384A CN201310693744.1A CN201310693744A CN103716384A CN 103716384 A CN103716384 A CN 103716384A CN 201310693744 A CN201310693744 A CN 201310693744A CN 103716384 A CN103716384 A CN 103716384A
Authority
CN
China
Prior art keywords
unit
cloud storage
data
storage system
metadata node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201310693744.1A
Other languages
Chinese (zh)
Inventor
高风朴
黄钦皇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CHUANGXINKE SOFTWARE TECHNOLOGY (SHENZHEN) Co Ltd
UIT STORAGE TECHNOLOGY (SHENZHEN) Co Ltd
Original Assignee
CHUANGXINKE SOFTWARE TECHNOLOGY (SHENZHEN) Co Ltd
UIT STORAGE TECHNOLOGY (SHENZHEN) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CHUANGXINKE SOFTWARE TECHNOLOGY (SHENZHEN) Co Ltd, UIT STORAGE TECHNOLOGY (SHENZHEN) Co Ltd filed Critical CHUANGXINKE SOFTWARE TECHNOLOGY (SHENZHEN) Co Ltd
Priority to CN201310693744.1A priority Critical patent/CN103716384A/en
Publication of CN103716384A publication Critical patent/CN103716384A/en
Pending legal-status Critical Current

Links

Abstract

The application provides a method and device for realizing cloud storage data synchronization in a cross-data-center manner. In the invention, a cloud storage system is established in a source data center and a target data center respectively, and the maximum number of per data unit is acquired from a metadata node local of a source cloud storage system and the metadata node local of a target cloud storage system respectively. The maximum numbers of per data unit acquired from the metadata node local of the source cloud storage system and the metadata node local of the target cloud storage system are compared, and when the former is greater than the latter, per data unit meeting the following conditions is acquired from the metadata node local of the source cloud storage system: when the number is greater than the one acquired from the metadata node local of the target cloud storage system, data synchronization is performed on the acquired per data unit between the source data center and the target data center. Therefore, cloud storage data synchronization in the cross-data-center manner is realized and high reliability of cloud storage is guaranteed.

Description

Across data center, realize the synchronous method and apparatus of cloud storage data
Technical field
The application relates to computer memory technical, particularly across data center, realizes the synchronous method and apparatus of cloud storage data.
Background technology
Cloud is stored in cloud computing (Cloud Computing) conceptive extension and is developed out.Cloud computing is the development of distributed treatment (Distributed Computing), parallel processing (Parallel Computing) and grid computing (Grid Computing), be to see through network huge computing program is split into numerous less subprogram automatically, then transfer to bulky systems that multi-section server forms result to be returned to user after analyzing as calculated.By cloud computing technology, Internet Service Provider can be within the several seconds, processes number in hundred million information necessarily even, reach and " supercomputer " same powerful network service.
Concept and the cloud computing of cloud storage are similar, it refers to by functions such as cluster application, grid or distributed file systems, a large amount of various dissimilar memory devices in network are gathered to collaborative work by application software, the common system that data storage and Operational Visit function are externally provided, guarantee the fail safe of data, and save storage sky.
At present, cloud is stored a kind of trend that has become storage development in future, and still, the development trend of following cloud storage, mainly still develops from fail safe, reliability, practicality equal angles.It is synchronous that Er Kua data center realizes cloud storage data, met the requirement that cloud is stored in high reliability, and it breaks down such as power down is even in the situation of the disaster such as massive blackout Yi data center, still can recover from another data center.
Therefore, across data center, realizing the synchronous method of cloud storage data is current technical problem urgently to be resolved hurrily.
Summary of the invention
The application provides Liao Kua data center to realize the synchronous method and apparatus of cloud storage data, and it is synchronous that Yi Kua data center realizes cloud storage data, guarantees the high reliability of cloud storage.
The technical scheme that the application provides comprises:
Across data center, realize the synchronous method of cloud storage data, comprising:
Respectively at source data center, cloud storage system is set up at destination data center, described source data center, two of offices, destination data center diverse location, at source data center, the cloud storage system that destination data center is set up all comprises metadata node, the cloud storage system of setting up at described source data center, be called source cloud storage system, for cloud stores service is externally provided, when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality, the cloud storage system of setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system,
From metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain respectively the maximum numbering of unit data unit, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point;
Relatively from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain the maximum numbering of unit data unit,
If the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered;
If the former is less than the latter, send alarm prompt;
If the former is greater than the latter, from metadata node this locality of described source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
A kind ofly across data center, realize the synchronous device of cloud storage data, this device is used at source data center, between destination data center, carry out data synchronous, wherein, described source data center, two of offices, destination data center diverse location, and all set up cloud storage system, at source data center, the cloud storage system that destination data center is set up all comprises metadata node, the cloud storage system of setting up at described source data center, be called source cloud storage system, for cloud stores service is externally provided, when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality, the cloud storage system of setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system,
This device comprises:
Acquiring unit, for obtain respectively the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point;
Comparing unit, for relatively obtain the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system,
If the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered;
If the former is less than the latter, send alarm prompt;
If the former is greater than the latter, send notice to data synchronisation unit;
Described data synchronisation unit, for obtaining from metadata node this locality of described source cloud storage system the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
As can be seen from the above technical solutions, in the present invention, by respectively at source data center, cloud storage system is set up at destination data center, from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain respectively the maximum numbering of unit data unit, relatively from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain the maximum numbering of unit data unit, when the former is greater than the latter, from metadata node this locality of described source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, at described source data center, the unit data unit executing data between destination data center, this being obtained is synchronous.This realizes Liao Kua data center, and to realize cloud storage data synchronous, guarantees the high reliability of cloud storage.
Accompanying drawing explanation
The method flow diagram that Fig. 1 provides for the embodiment of the present invention;
The step 104 realization flow figure that Fig. 2 provides for the embodiment of the present invention;
The synchronous flow chart of unit data unit executing data between described source data center, destination data center, this being obtained that Fig. 3 provides for the embodiment of the present invention;
The structure drawing of device that Fig. 4 provides for the embodiment of the present invention.
Embodiment
In order to make the object, technical solutions and advantages of the present invention clearer, below in conjunction with the drawings and specific embodiments, describe the present invention.
The invention provides across data center and realize the synchronous method of cloud storage data, below by Fig. 1, be described:
Referring to Fig. 1, the method flow diagram that Fig. 1 provides for the embodiment of the present invention.As shown in Figure 1, this flow process can comprise the following steps:
Step 101, respectively at source data center, destination data center sets up cloud storage system.
Here, source data center, two of offices, destination data center diverse location, two places are apart from possibility hundreds of kilometer, even thousands of kilometers.
In the present invention, at source data center, destination data center sets up cloud storage system, its structure is similar, mainly comprise: metadata node and Chunk node, wherein, metadata node is used for managing cloud storage system, mainly stored the index information of some files, and Chunk node has been stored the particular content of file.
In the present invention, for ease of describing, by the cloud storage system of setting up at described source data center, be called source cloud storage system, this source cloud storage system externally provides cloud stores service under normal circumstances, be specially: when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality; Here, described Data Update request generally comprises: write request and/or removal request.Such as, when the metadata node of source cloud storage system receives write request, related data corresponding to this write request is organized into a unit data unit (being called metalog), the mode increasing progressively according to number order is that described unit data unit distributes numbering, such as, before not receiving this write request, the maximum that local unit data unit has distributed is numbered M, and when receiving write request, related data corresponding to this write request is organized into a unit data unit, for distributing numbering M+1 in this unit data unit.Afterwards, described unit data unit and the numbering that is assigned with are stored in to this locality.
As one embodiment of the present of invention, related data corresponding to described Data Update request comprises but is not limited to:
Data Identification, described Data Identification is: the sign of file or file;
Action type; Wherein, described action type comprises: revise, delete or create.
Wherein, during the sign that is file at Data Identification, related data corresponding to described Data Update request also comprises the routing information that the content in file is stored at source cloud storage service system.
In the present invention, for ease of describing, the cloud storage system by setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system.
Step 102, from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain respectively the maximum numbering of unit data unit, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point.
As one embodiment of the present of invention, the maximum numbering that in step 102, metadata node this locality local from the metadata node of described source cloud storage system and described object cloud storage system obtains unit data unit is respectively that at destination data center, to set up object cloud storage system be to carry out setting-up time foundation after source cloud storage system is set up at source data center and described source cloud storage system complete Data Update according to Data Update request.Here, described setting-up time is that source cloud storage system and object cloud storage system are set up the time of being separated by.
And if the object cloud storage system that destination data center is set up is set up after exceeding described setting-up time; can not perform step 102; directly source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center.Reason is: the object cloud storage system that destination data center is set up is set up after exceeding described setting-up time; illustrate that source cloud storage system is by may having stored a large amount of unit datas unit to current; this can cause last between described source data center, destination data center the synchronous unit data unit of executing data many; for saving lock in time; source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center.
Step 103, relatively from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain the maximum numbering of unit data unit, if the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered; If the former is less than the latter, send alarm prompt; If the former is greater than the latter, perform step 104.
Step 104, from metadata node this locality of described source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
As one embodiment of the present of invention, preferably, in this step 104, described metadata node this locality from described source cloud storage system obtains numbering that metadata node metadata node this locality local and described object cloud storage system of Shi Cong source, the unit data unit cloud storage system satisfying condition obtains and differs and carry out the prerequisite within setting range.
And if the numbering that metadata node this locality of and described object cloud storage system local from the metadata node of source cloud storage system obtains differs over setting range; do not perform step 104; but source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center.Reason is: if the numbering that metadata node this locality of and described object cloud storage system local from the metadata node of source cloud storage system obtains differs over setting range; explanation last between described source data center, destination data center the synchronous unit data unit of executing data many; for saving lock in time; source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center.
So far, complete flow process shown in Fig. 1.Preferably, in flow process shown in Fig. 1 step 104 complete data synchronous after, can also further this be participated in to the number record of unit data unit of synchronous maximum numbering at object cloud storage server, so that the numbering of the maximum in real time inspection destination server.
In flow process shown in Fig. 1, as a preferred embodiment of the present invention, metadata node this locality from source cloud storage system described in step 104 obtains the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained synchronously can pass through shown in Fig. 2:
Referring to Fig. 2, the step 104 realization flow figure that Fig. 2 provides for the embodiment of the present invention.As shown in Figure 2, this flow process can comprise the following steps:
Step 201, identifies the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system.
Step 202, relatively the batch unit N of this unit data element number identifying and setting, if the former is less than or equal to the latter, performs step 203, if the former is greater than the latter, performs step 204.
Here, N can be according to actual conditions setting, such as being set to 50.
Step 203, directly from metadata node this locality of source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
Step 204, first from metadata node this locality of source cloud storage system, obtain N the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous; Afterwards the maximum numbering in the unit data unit obtaining is numbered as candidate, identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than candidate's numbering, return to the relatively operation of the batch unit N of this unit data element number identifying and setting.
This step 204 is to carry out under the prerequisite of the unit data element number that identifies at this batch unit N that is greater than setting.Take N as 50 being example, in this step 204, if that obtains from metadata node this locality of described object cloud storage system is numbered 50, that from metadata node this locality of described source cloud storage system, obtains is numbered 200, first from metadata node this locality of source cloud storage system, obtain the unit data unit that is numbered 51 to 100, the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous; Afterwards, from metadata node this locality of source cloud storage system, obtain the unit data unit that is numbered 101 to 150 again, the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous, finally, from metadata node this locality of source cloud storage system, obtain the unit data unit that is numbered 151 to 200, the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.By this in batches synchronous, can avoid when data are synchronous, occurring between source data center, destination data center network congestion.
So far, complete flow process shown in Fig. 2.
Wherein, in above-mentioned steps 203, step 204, the unit data unit executing data between described source data center, destination data center, this being obtained synchronously can be realized by flow process shown in Fig. 3:
Referring to Fig. 3, the synchronous flow chart of unit data unit executing data between described source data center, destination data center, this being obtained that Fig. 3 provides for the embodiment of the present invention.As shown in Figure 3, this flow process can comprise the following steps:
Step 301, for the per unit data cell of obtaining, identifies the Data Identification that this unit data unit comprises, and the sign of file, performs step 302 if, and file identification, performs step 303 if.
Step 302, all Organization of Datas that directly this unit data unit comprised become a message to send to destination data center.
Step 303, all Organization of Datas that this unit data unit is comprised become a message to send to destination data center, and the action type comprising in this unit data unit is when deleting, the routing information comprising according to this unit data unit obtains file content, and the file content obtaining is write to destination data center.
So far, complete flow process shown in Fig. 3.By flow process shown in Fig. 3, can realize that across data center, to realize cloud storage data synchronous, guarantee the high reliability of cloud storage.
It should be noted that, in the present invention, before the unit data unit executing data between source data center, destination data center, this being obtained is synchronous, the method further comprises:
Dedicated bandwidth based on setting is set up network and is connected between source data center, destination data center,
If network connects, be not successfully established, attempt set point number, after attempting set point number, still the unsuccessful network of successfully setting up between source data center, destination data center connects, and sends alarm prompt.
Above method provided by the invention is described, below device provided by the invention is described:
Referring to Fig. 4, the structure drawing of device that Fig. 4 provides for the embodiment of the present invention.This device is used at source data center, between destination data center, carry out data synchronous, wherein, described source data center, two of offices, destination data center diverse location, and all set up cloud storage system, at source data center, the cloud storage system that destination data center is set up all comprises metadata node, the cloud storage system of setting up at described source data center, be called source cloud storage system, for cloud stores service is externally provided, when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality, the cloud storage system of setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system,
As shown in Figure 4, this device comprises:
Acquiring unit, for obtain respectively the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point;
Comparing unit, for relatively obtain the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system,
If the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered;
If the former is less than the latter, send alarm prompt;
If the former is greater than the latter, send notice to data synchronisation unit;
Described data synchronisation unit, for obtaining from metadata node this locality of described source cloud storage system the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
Preferably, in the present invention, related data corresponding to described Data Update request comprises but is not limited to:
Data Identification, described Data Identification is: the sign of file or file;
During the sign that is file at Data Identification, the content in file is at the routing information of source cloud storage service system storage; During the sign that is file at Data Identification, file is at the routing information of source cloud storage service system storage;
Action type; Wherein, described action type comprises: revise, delete or create;
There is the timestamp of operation.
Preferably, in the present invention, described acquiring unit obtains from metadata node this locality of source cloud storage system the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained synchronously comprises:
Identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system;
The batch unit N of this unit data element number identifying and setting relatively;
If the former is less than or equal to the latter, directly from metadata node this locality of source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous;
If the former is greater than the latter, first from metadata node this locality of source cloud storage system, obtain N the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous; Afterwards the maximum numbering in the unit data unit obtaining is numbered as candidate, identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than candidate's numbering, return to the relatively operation of the batch unit N of this unit data element number identifying and setting.
Preferably, in the present invention, the unit data unit executing data that described data synchronisation unit is obtained this between source data center, destination data center synchronously comprises:
For the per unit data cell of obtaining, if the sign that the Data Identification that this unit data unit comprises is file, all Organization of Datas that directly this unit data unit comprised become a message to send to destination data center; If the sign that the Data Identification that this unit data unit comprises is file, the all Organization of Datas that this unit data unit comprised become a message to send to destination data center, and the action type comprising in this unit data unit is when deleting, the routing information comprising according to this unit data unit obtains file content, and the file content obtaining is write to destination data center.
Preferably, in the present invention, the described acquiring unit maximum numbering that metadata node this locality local from the metadata node of described source cloud storage system and described object cloud storage system obtains unit data unit is respectively that at destination data center, to set up object cloud storage system be to carry out setting-up time foundation after source cloud storage system is set up at source data center and described source cloud storage system complete Data Update according to Data Update request;
Described data synchronisation unit is obtained numbering that metadata node metadata node this locality local and described object cloud storage system of Shi Cong source, the unit data unit cloud storage system satisfying condition obtains and is differed and carry out the prerequisite within setting range from metadata node this locality of described source cloud storage system;
The object cloud storage system that described data synchronisation unit is also set up at destination data center is set up after exceeding setting-up time, or, when the numbering that metadata node this locality local from the metadata node of source cloud storage system and described object cloud storage system obtains differs over setting range, further source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center; Described setting-up time is that source cloud storage system and object cloud storage system are set up the time of being separated by.
So far, complete the unit describe shown in Fig. 4.
As can be seen from the above technical solutions, in the present invention, by respectively at source data center, cloud storage system is set up at destination data center, from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain respectively the maximum numbering of unit data unit, relatively from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain the maximum numbering of unit data unit, when the former is greater than the latter, from metadata node this locality of described source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, at described source data center, the unit data unit executing data between destination data center, this being obtained is synchronous.This realizes Liao Kua data center, and to realize cloud storage data synchronous, guarantees the high reliability of cloud storage.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any modification of making, be equal to replacement, improvement etc., within all should being included in the scope of protection of the invention.

Claims (13)

1.Yi Zhongkua data center realizes the synchronous method of cloud storage data, it is characterized in that, the method comprises:
Respectively at source data center, cloud storage system is set up at destination data center, described source data center, two of offices, destination data center diverse location, at source data center, the cloud storage system that destination data center is set up all comprises metadata node, the cloud storage system of setting up at described source data center, be called source cloud storage system, for cloud stores service is externally provided, when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality, the cloud storage system of setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system,
From metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain respectively the maximum numbering of unit data unit, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point;
Relatively from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, obtain the maximum numbering of unit data unit,
If the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered;
If the former is less than the latter, send alarm prompt;
If the former is greater than the latter, from metadata node this locality of described source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
2. method according to claim 1, is characterized in that, related data corresponding to described Data Update request comprises but be not limited to:
Data Identification, described Data Identification is: the sign of file or file;
During the sign that is file at Data Identification, the content in file is at the routing information of source cloud storage service system storage; During the sign that is file at Data Identification, file is at the routing information of source cloud storage service system storage;
Action type; Wherein, described action type comprises: revise, delete or create;
There is the timestamp of operation.
3. method according to claim 1 and 2, it is characterized in that, described metadata node this locality from source cloud storage system obtains the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained synchronously comprises:
Identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system;
The batch unit N of this unit data element number identifying and setting relatively;
If the former is less than or equal to the latter, directly from metadata node this locality of source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous;
If the former is greater than the latter, first from metadata node this locality of source cloud storage system, obtain N the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous; Afterwards the maximum numbering in the unit data unit obtaining is numbered as candidate, identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than candidate's numbering, return to the relatively operation of the batch unit N of this unit data element number identifying and setting.
4. method according to claim 3, is characterized in that, the described unit data unit executing data between source data center, destination data center, this being obtained synchronously comprises:
For the per unit data cell of obtaining, if the sign that the Data Identification that this unit data unit comprises is file, all Organization of Datas that directly this unit data unit comprised become a message to send to destination data center; If the sign that the Data Identification that this unit data unit comprises is file, the all Organization of Datas that this unit data unit comprised become a message to send to destination data center, and the action type comprising in this unit data unit is when deleting, the routing information comprising according to this unit data unit obtains file content, and the file content obtaining is write to destination data center.
5. method according to claim 1, it is characterized in that, the maximum numbering that described metadata node this locality local from the metadata node of described source cloud storage system and described object cloud storage system respectively obtains unit data unit is that at destination data center, to set up object cloud storage system be to carry out setting-up time foundation after source cloud storage system is set up at source data center and described source cloud storage system complete Data Update according to Data Update request.
6. method according to claim 1, it is characterized in that, described metadata node this locality from described source cloud storage system obtains numbering that metadata node metadata node this locality local and described object cloud storage system of Shi Cong source, the unit data unit cloud storage system satisfying condition obtains and differs and carry out the prerequisite within setting range.
7. according to the method described in claim 5 or 6, it is characterized in that, if the object cloud storage system that destination data center is set up is set up after exceeding setting-up time, or, the numbering that metadata node this locality local from the metadata node of source cloud storage system and described object cloud storage system obtains differs over setting range, and the method further comprises:
Source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center;
Described setting-up time is that source cloud storage system and object cloud storage system are set up the time of being separated by.
8. method according to claim 1, is characterized in that, before the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous, the method further comprises:
Dedicated bandwidth based on setting is set up network and is connected between source data center, destination data center,
If network connects, be not successfully established, attempt set point number, after attempting set point number, still the unsuccessful network of successfully setting up between source data center, destination data center connects, and sends alarm prompt.
9.Yi Zhongkua data center realizes the synchronous device of cloud storage data, it is characterized in that, this device is used at source data center, between destination data center, carry out data synchronous, wherein, described source data center, two of offices, destination data center diverse location, and all set up cloud storage system, at source data center, the cloud storage system that destination data center is set up all comprises metadata node, the cloud storage system of setting up at described source data center, be called source cloud storage system, for cloud stores service is externally provided, when receiving Data Update request, related data corresponding to described Data Update request is organized into a unit data unit, the mode increasing progressively according to number order is that described unit data unit distributes numbering, and described unit data unit and the numbering that is assigned with are stored in to this locality, the cloud storage system of setting up at described destination data center, is called object cloud storage system, for backup source cloud storage system,
This device comprises:
Acquiring unit, for obtain respectively the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system, wherein, if metadata node this locality of object cloud storage system is storage cell data cell not, the maximum of giving tacit consent to the local unit data of the metadata node unit of object cloud storage system is numbered set point;
Comparing unit, for relatively obtain the maximum numbering of unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system,
If the former equals the latter, wait for setting-up time, return to afterwards respectively the operation that the maximum of obtaining unit data unit from metadata node metadata node this locality local and described object cloud storage system of described source cloud storage system is numbered;
If the former is less than the latter, send alarm prompt;
If the former is greater than the latter, send notice to data synchronisation unit;
Described data synchronisation unit, for obtaining from metadata node this locality of described source cloud storage system the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous.
10. device according to claim 9, is characterized in that, related data corresponding to described Data Update request comprises but be not limited to:
Data Identification, described Data Identification is: the sign of file or file;
During the sign that is file at Data Identification, the content in file is at the routing information of source cloud storage service system storage; During the sign that is file at Data Identification, file is at the routing information of source cloud storage service system storage;
Action type; Wherein, described action type comprises: revise, delete or create;
There is the timestamp of operation.
11. according to the device described in claim 9 or 10, it is characterized in that, described acquiring unit obtains from metadata node this locality of source cloud storage system the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained synchronously comprises:
Identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system;
The batch unit N of this unit data element number identifying and setting relatively;
If the former is less than or equal to the latter, directly from metadata node this locality of source cloud storage system, obtain the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous;
If the former is greater than the latter, first from metadata node this locality of source cloud storage system, obtain N the unit data unit meeting the following conditions: numbering is greater than the numbering of obtaining from metadata node this locality of described object cloud storage system, and the unit data unit executing data between described source data center, destination data center, this being obtained is synchronous; Afterwards the maximum numbering in the unit data unit obtaining is numbered as candidate, identify the local unit data element number meeting the following conditions existing of metadata node of described source cloud storage system: numbering is greater than candidate's numbering, return to the relatively operation of the batch unit N of this unit data element number identifying and setting.
12. devices according to claim 11, is characterized in that, the unit data unit executing data that described data synchronisation unit is obtained this between source data center, destination data center synchronously comprises:
For the per unit data cell of obtaining, if the sign that the Data Identification that this unit data unit comprises is file, all Organization of Datas that directly this unit data unit comprised become a message to send to destination data center; If the sign that the Data Identification that this unit data unit comprises is file, the all Organization of Datas that this unit data unit comprised become a message to send to destination data center, and the action type comprising in this unit data unit is when deleting, the routing information comprising according to this unit data unit obtains file content, and the file content obtaining is write to destination data center.
13. devices according to claim 9, it is characterized in that, the described acquiring unit maximum numbering that metadata node this locality local from the metadata node of described source cloud storage system and described object cloud storage system obtains unit data unit is respectively that at destination data center, to set up object cloud storage system be to carry out setting-up time foundation after source cloud storage system is set up at source data center and described source cloud storage system complete Data Update according to Data Update request;
Described data synchronisation unit is obtained numbering that metadata node metadata node this locality local and described object cloud storage system of Shi Cong source, the unit data unit cloud storage system satisfying condition obtains and is differed and carry out the prerequisite within setting range from metadata node this locality of described source cloud storage system;
The object cloud storage system that described data synchronisation unit is also set up at destination data center is set up after exceeding setting-up time, or, when the numbering that metadata node this locality local from the metadata node of source cloud storage system and described object cloud storage system obtains differs over setting range, further source cloud storage system is shut down, and synchronous according to the mode of batch synchronization executing data between source data center, destination data center; Described setting-up time is that source cloud storage system and object cloud storage system are set up the time of being separated by.
CN201310693744.1A 2013-12-17 2013-12-17 Method and device for realizing cloud storage data synchronization in cross-data-center manner Pending CN103716384A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310693744.1A CN103716384A (en) 2013-12-17 2013-12-17 Method and device for realizing cloud storage data synchronization in cross-data-center manner

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310693744.1A CN103716384A (en) 2013-12-17 2013-12-17 Method and device for realizing cloud storage data synchronization in cross-data-center manner

Publications (1)

Publication Number Publication Date
CN103716384A true CN103716384A (en) 2014-04-09

Family

ID=50408954

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310693744.1A Pending CN103716384A (en) 2013-12-17 2013-12-17 Method and device for realizing cloud storage data synchronization in cross-data-center manner

Country Status (1)

Country Link
CN (1) CN103716384A (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104168335A (en) * 2014-09-02 2014-11-26 东软熙康健康科技有限公司 Data synchronization method and device
CN105554135A (en) * 2015-12-23 2016-05-04 北京奇虎科技有限公司 Method and device for synchronizing meta information among digital nodes
CN106921703A (en) * 2015-12-25 2017-07-04 阿里巴巴集团控股有限公司 The method of cross-border data syn-chronization, system, and domestic and overseas data center
CN108121804A (en) * 2017-12-22 2018-06-05 百度在线网络技术(北京)有限公司 Method, apparatus, terminal and the storage medium of cross-region distributed storage data
CN108259543A (en) * 2016-12-29 2018-07-06 广东中科遥感技术有限公司 Distributed cloud storage database and its be deployed in the method for multiple data centers
CN108475254A (en) * 2016-12-16 2018-08-31 华为技术有限公司 Object clone method, device and object storage device
CN109726250A (en) * 2018-12-27 2019-05-07 星环信息科技(上海)有限公司 Data-storage system, metadatabase synchronization and data cross-domain calculation method
WO2022078243A1 (en) * 2020-10-16 2022-04-21 北京沃东天骏信息技术有限公司 Information synchronization method and apparatus

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1489052A (en) * 2002-10-11 2004-04-14 鸿富锦精密工业(深圳)有限公司 Multi-node file syn chronizing system and method
CN1984461A (en) * 2006-04-07 2007-06-20 华为技术有限公司 Method for restoring user information by access position register
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN101808137A (en) * 2010-03-29 2010-08-18 成都市华为赛门铁克科技有限公司 Data transmission method, device and system
CN102073494A (en) * 2010-12-30 2011-05-25 用友软件股份有限公司 Method and device for managing cache data
CN102693168A (en) * 2011-03-22 2012-09-26 中兴通讯股份有限公司 A method, a system and a service node for data backup recovery
US20130218837A1 (en) * 2012-02-16 2013-08-22 Verizon Patent And Licensing Inc. Cloud data synchronization with local data

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1489052A (en) * 2002-10-11 2004-04-14 鸿富锦精密工业(深圳)有限公司 Multi-node file syn chronizing system and method
CN1984461A (en) * 2006-04-07 2007-06-20 华为技术有限公司 Method for restoring user information by access position register
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN101808137A (en) * 2010-03-29 2010-08-18 成都市华为赛门铁克科技有限公司 Data transmission method, device and system
CN102073494A (en) * 2010-12-30 2011-05-25 用友软件股份有限公司 Method and device for managing cache data
CN102693168A (en) * 2011-03-22 2012-09-26 中兴通讯股份有限公司 A method, a system and a service node for data backup recovery
US20130218837A1 (en) * 2012-02-16 2013-08-22 Verizon Patent And Licensing Inc. Cloud data synchronization with local data

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104168335A (en) * 2014-09-02 2014-11-26 东软熙康健康科技有限公司 Data synchronization method and device
CN104168335B (en) * 2014-09-02 2018-08-03 宁波(东软熙康)智慧健康研究院有限公司 A kind of method of data synchronization and device
CN105554135A (en) * 2015-12-23 2016-05-04 北京奇虎科技有限公司 Method and device for synchronizing meta information among digital nodes
CN105554135B (en) * 2015-12-23 2019-03-05 北京奇虎科技有限公司 The method and device of synchronous metamessage between back end
CN106921703A (en) * 2015-12-25 2017-07-04 阿里巴巴集团控股有限公司 The method of cross-border data syn-chronization, system, and domestic and overseas data center
CN108475254A (en) * 2016-12-16 2018-08-31 华为技术有限公司 Object clone method, device and object storage device
CN108259543B (en) * 2016-12-29 2021-07-06 广东中科遥感技术有限公司 Distributed cloud storage database and method for deploying same in multiple data centers
CN108259543A (en) * 2016-12-29 2018-07-06 广东中科遥感技术有限公司 Distributed cloud storage database and its be deployed in the method for multiple data centers
CN108121804A (en) * 2017-12-22 2018-06-05 百度在线网络技术(北京)有限公司 Method, apparatus, terminal and the storage medium of cross-region distributed storage data
CN108121804B (en) * 2017-12-22 2020-06-05 百度在线网络技术(北京)有限公司 Cross-region distributed data storage method, device, terminal and storage medium
CN109726250B (en) * 2018-12-27 2020-01-17 星环信息科技(上海)有限公司 Data storage system, metadata database synchronization method and data cross-domain calculation method
CN109726250A (en) * 2018-12-27 2019-05-07 星环信息科技(上海)有限公司 Data-storage system, metadatabase synchronization and data cross-domain calculation method
WO2022078243A1 (en) * 2020-10-16 2022-04-21 北京沃东天骏信息技术有限公司 Information synchronization method and apparatus

Similar Documents

Publication Publication Date Title
US20220206889A1 (en) Automatic correlation of dynamic system events within computing devices
CN103716384A (en) Method and device for realizing cloud storage data synchronization in cross-data-center manner
US10795905B2 (en) Data stream ingestion and persistence techniques
US20150134797A1 (en) Managed service for acquisition, storage and consumption of large-scale data streams
CN103473277B (en) The Snapshot Method and device of file system
CN109034993B (en) Account checking method, account checking equipment, account checking system and computer readable storage medium
US20190042659A1 (en) Data writing and reading and apparatus and cloud storage system
US8375200B2 (en) Embedded device and file change notification method of the embedded device
US8768929B2 (en) Clustering streaming graphs
CN108234170B (en) Monitoring method and device for server cluster
CN108108127B (en) File reading method and system
AU2014209697A1 (en) Method and system for using a recursive event listener on a node in hierarchical data structure
US20150006459A1 (en) Alarm Correlation Analysis Method, Apparatus and System
CN111309550A (en) Data acquisition method, system, equipment and storage medium of application program
CN106649676B (en) HDFS (Hadoop distributed File System) -based duplicate removal method and device for stored files
WO2016029744A1 (en) Metadata recovery method and relevant device
CN103186418B (en) A kind of task distribution method and system
US10331484B2 (en) Distributed data platform resource allocator
CN104462342B (en) database snapshot synchronization processing method and device
CN109144785B (en) Method and apparatus for backing up data
CN111061719A (en) Data collection method, device, equipment and storage medium
US11386043B2 (en) Method, device, and computer program product for managing snapshot in application environment
CN113609091B (en) Log management method, device, equipment and storage medium
CN109522152B (en) Disaster recovery method, storage medium and device for electron microscope data
CN105511952A (en) Resource self-migration method and system based on cloud computing platform

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20140409

RJ01 Rejection of invention patent application after publication