CN101986276B - Methods and systems for storing and recovering files and server - Google Patents
Methods and systems for storing and recovering files and server Download PDFInfo
- Publication number
- CN101986276B CN101986276B CN201010526834.8A CN201010526834A CN101986276B CN 101986276 B CN101986276 B CN 101986276B CN 201010526834 A CN201010526834 A CN 201010526834A CN 101986276 B CN101986276 B CN 101986276B
- Authority
- CN
- China
- Prior art keywords
- storage nodes
- file
- dummy
- node
- object storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The embodiment of the invention discloses methods and systems for storing and recovering files and a server. The method for storing files comprises the following steps: after dividing the file to be stored into at least two object files, ensuring each object file to correspond to a virtual storage node; and creating at least two object sets on each object storage node and mapping each virtual storage node to at least two object sets, wherein at least two object sets belong to different object storage nodes; and at least two object sets are used for storing the object files corresponding to the mapped virtual storage nodes. The object files in the invention are not directly stored on the object storage nodes but an intermediate control layer is provided for storing the object files via the mapping relations between the virtual storage nodes and the object sets; and when the object storage nodes have failure, the corresponding mapping relations on the virtual storage nodes with storage relations with the object storage nodes only need to be maintained, thus being convenient for recovering the files and improving the recovery speeds of the files.
Description
Technical field
The present invention relates to communication technical field, particularly file memory method, file restoration methods, system and server.
Background technology
In parallel file system, generally include CA (Client Agent, Client Agent), MDS (Meta-data Server, meta data server) and OSN (Object Storage Node, object storage nodes).Wherein, CA is used to application program that user's development interface of visit parallel file system is provided; MDS is responsible for providing to client the metadata of total system, manage the NameSpace of total system, safeguard bibliographic structure and the user right of total system, and the data consistency of responsible maintenance system, MDS also is used for safeguarding the deployment information of OSN, provides the file deployment scheme to CA; OSN is used for object form storage data.File in the parallel file system is distributed on each OSN with the form of band, and a band can be regarded a data object as.CA holds the metadata information that reads file to MDS when reading data, obtain the content of each band of file at least two OSN according to metadata information.
In the prior art, when the hard disk on the some OSN in the parallel file system breaks down, client is difficult to go up sense data from this OSN usually, therefore adopt a plurality of hard disks by RAID (Redundant Array of Independent Disk at OSN, raid-array) the technology mode that is combined into virtual separate unit big capacity hard disk is carried out data redundancy, when a data object among the OSN damages, can read corresponding data by other hard disk from this OSN this locality, according to the relation of the RAID between hard disk, data are recovered out.
The inventor finds in the research process to prior art, the RAID mode of recovering data on the existing OSN is unit with the hard disk, increase along with single hard-disk capacity, and the number of times of hard disk per second input and output does not increase a lot, so in the hard disk data more for a long time, the release time of data will be elongated; And because on the other hand, the recovery of data only limits to local node OSN inside, if therefore OSN itself goes wrong, then can cause data to be difficult to recover.
Summary of the invention
The purpose of the embodiment of the invention is to provide file memory method, file restoration methods, system and server, and is long release time to solve existing OSN data when the fault, even the problem that is difficult to recover.
For solving the problems of the technologies described above, the invention provides following technical scheme:
A kind of file memory method comprises:
After file to be stored is divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node; Create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, described at least two object sets belong to different object storage nodes, and described at least two object sets are used for storing the obj ect file of the dummy storage node correspondence of shining upon; Preserve the mapping relations table between described dummy storage node, described object storage nodes and described object set;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
A kind of file restoration methods, the file that the application of aforementioned file memory method is stored recovers, and comprising:
Receive the file recovery request;
When comprising the information that object storage nodes breaks down in the described file recovery request, search the mapping relations table between dummy storage node, object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down;
Obtain the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon;
According to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down is recovered;
Upgrade the mapping relations table between described dummy storage node, object storage nodes and object set.
A kind of document storage system comprises: meta data server and at least two object storage nodes, wherein,
Described meta data server, after file to be stored is divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node, create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, and described at least two object sets belong to different object storage nodes; Preserve the mapping relations table between described dummy storage node, described object storage nodes and described object set;
Described object storage nodes is for the obj ect file of the dummy storage node correspondence of shining upon by the object set storage;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
A kind of file recovery system, the file that the application of aforementioned document storage system is stored recovers, and comprising: meta data server and at least two object storage nodes, wherein,
Described meta data server, be used for receiving the file recovery request, when comprising the information that object storage nodes breaks down in the described file recovery request, from dummy storage node, search in the mapping relations table between object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down, obtain the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon, according to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down is recovered, and upgrade described dummy storage node, mapping relations table between object storage nodes and object set.
A kind of server is used for file is stored, and comprising:
Division unit is used for file to be stored is divided at least two obj ect file;
Corresponding unit is used for each obj ect file is corresponded to a dummy storage node;
Creating unit is used for creating at least two object sets in each object storage nodes;
Map unit is used for each dummy storage node is mapped at least two object sets, and described at least two object sets belong to different object storage nodes, and described at least two object sets are used for storing the obj ect file of the dummy storage node correspondence of shining upon;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
A kind of server, the file of application of aforementioned server stores recovers, and comprising:
Receiving element is used for receiving the file recovery request;
Search the unit, be used for when described file recovery request comprises the information that object storage nodes breaks down, searching the mapping relations table between dummy storage node, object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down;
Acquiring unit is used for obtaining the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon;
Recovery unit is used for according to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down being recovered, and upgrades the mapping relations table between described dummy storage node, object storage nodes and object set.
The technical scheme that is provided by the above embodiment of the invention as seen, after in the embodiment of the invention file to be stored being divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node, create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, the obj ect file of the dummy storage node correspondence of shining upon by the storage of at least two object sets, at least two object sets belong to different object storage nodes; Obj ect file is not directly to be stored on the object storage nodes in the embodiment of the invention that hence one can see that, but by the mapping relations between dummy storage node and the object set, for the storage of obj ect file provide one in the middle of key-course, therefore receive when comprising the recovery request that object storage nodes breaks down, search the object storage nodes corresponding virtual memory node that breaks down with this, obtain the object storage nodes except the object storage nodes that breaks down that each dummy storage node of finding is shone upon, according to the object set on the object storage nodes of obtaining the object set on the object storage nodes that breaks down is recovered; Because when object storage nodes breaks down, only needing to safeguard with this object storage nodes has corresponding mapping relations on the dummy storage node of the relation of depositing, therefore is convenient to the recovery of file, has improved the speed that file recovers.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, to do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below, apparently, the accompanying drawing that describes below only is some embodiment that put down in writing among the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the embodiment process flow diagram of file memory method of the present invention;
Fig. 2 is the structural representation of the applied a kind of file storage of the embodiment of the invention;
Fig. 3 is the first embodiment process flow diagram of file restoration methods of the present invention;
Fig. 4 is the second embodiment process flow diagram of file restoration methods of the present invention;
Fig. 5 is the 3rd embodiment process flow diagram of file restoration methods of the present invention;
Fig. 6 is the embodiment block diagram of system of the present invention;
Fig. 7 is an embodiment block diagram of server of the present invention;
Fig. 8 is another embodiment block diagram of server of the present invention.
Embodiment
The embodiment of the invention provides file memory method, file restoration methods, system and server.
In order to make those skilled in the art person understand the present invention program better, and above-mentioned purpose of the present invention, feature and advantage can be become apparent more, the present invention is further detailed explanation below in conjunction with the drawings and specific embodiments.
Referring to Fig. 1, be the embodiment process flow diagram of file memory method of the present invention:
Step 101: file to be stored is divided at least two obj ect file.
In advance, comprise at least two object storage nodes in the system, create at least two object sets in each object storage nodes, object set at least two object storage nodes is made up, the object set of each described combination is corresponding with a dummy storage node, is about to a dummy storage node and is mapped on the different object sets of at least two different object storage nodes.
Operate requirement for the concurrent reading and writing to file in the realization system, usually a file is divided at least two obj ect file and stores, in order to read while write obj ect file, reduce the time of file processing.
Step 102: each obj ect file is corresponded to a dummy storage node.
In the embodiment of the invention, obj ect file is not directly to be stored on the different object storage nodes, but each obj ect file corresponds on the dummy storage node, and namely obj ect file and dummy storage node are the relations of shining upon one by one.
Step 103: create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, at least two object sets belong to different object storage nodes.
What in the embodiment of the invention obj ect file is stored is object set on the object storage nodes, each dummy node shines upon mutually with different object sets at least two object storage nodes, all are mapped to the identical obj ect file of object centralized stores of a dummy storage node, backup mutually is convenient to file is recovered between these object sets.
Step 104: the obj ect file of the dummy storage node correspondence of shining upon by the storage of at least two object sets, finish current flow process.
Further, determine main object storage nodes according to the storage capacity of described at least two object storage nodes and from object storage nodes.In addition, can also regularly monitor the memory capacity of each dummy storage node, when memory capacity surpasses the threshold value set in advance, create new object set in object storage nodes, and be mapped to new dummy storage node after new object set made up.
Object set in the embodiment of the invention (Object Set is called for short Objset) refers to the set of a group objects, and two real object sets are combined, and is mapped as a vOSN (virtual OSN, virtual OSN).In the prior art, do not have the concept of object set, a file is mapped to 3 objects, and each object leaves on the OSN node, and this file is safeguarded by a table to the mapping relations unification of object.Need carry out file when recovering when certain OSN node damages, then all therewith OSN have the description information of files of the relation of depositing all will write again one time, when number of files was very big, the index word that relates to was also very big; The concept of embodiment of the invention application collection, a file is mapped to 3 objects, and each object leaves on the vOSN, and a vOSN is mapped to again on two different OSN, by above-mentioned two-layer mapping relations.Need carry out file when recovering when certain OSN node damages, then only need to rewrite all therewith OSN the descriptor of the vOSN of the relation of depositing is arranged, do not need to change the information of the file of enormous amount, so index word is less relatively.
Referring to Fig. 2, be the applied a kind of file storage organization synoptic diagram of the embodiment of the invention.
Among Fig. 2, MDS is meta data server, is responsible for providing to client the metadata of total system, the bibliographic structure of management total system, the data consistency of user right and maintenance system; OSN is object storage nodes, is used for the form storage object file with object set.Comprised two-layer mapping relations in the file storage organization shown in Fig. 2, File-1 is example with storage, wherein:
The ground floor mapping relations are the mapping relations between file and the dummy storage node, comprising:
File-1---------><vOSN_1，Object_1>，<vOSN_2，Object_1>，<vOSN_3，Object_1>
Above-mentioned mapping relations are interpreted as, and File-1 is divided into three obj ect file, and each obj ect file corresponds on the dummy storage node, and namely three obj ect file correspond to dummy storage node vOSN_1, vOSN_2 and vOSN_3 respectively;
Second layer mapping relations are the mapping relations between the object set of dummy storage node and object storage nodes:
vOSN_1---------><OSN_1，Objset_1>，<OSN_2，Objset_1>
vOSN_2---------><OSN_1，Objset_2>，<OSN_3，Objset_1>
vOSN_3---------><OSN_2，Objset_2>，<OSN_3，Objset_2>
Above-mentioned mapping relations are interpreted as, dummy storage node vOSN_1 is mapped to the object set Objset_1 of object storage nodes OSN_1 respectively, and the object set Objset_1 of object storage nodes OSN_2, namely the obj ect file of a dummy storage node correspondence is stored in respectively in the object set of two mutually redundant object storage nodes; In like manner, dummy storage node vOSN_2 is mapped to the object set Objset_2 of object storage nodes OSN_1 respectively, and the object set Objset_1 of object storage nodes OSN_3, dummy storage node vOSN_3 is mapped to the object set Objset_2 of object storage nodes OSN_2 respectively, and the object set Objset_2 of object storage nodes OSN_3.
According to above-mentioned two-layer mapping relations, then the final mapping relations of File-1 and obj ect file are as follows:
File-1--------->
<OSN_1，Objset_1，Object_1>，<OSN_2，Objset_1，Object_1>
<OSN_1，Objset_2，Object_1>，<OSN_3，Objset_1，Object_1>
<OSN_2，Objset_2，Object_1>，<OSN_3，Objset_2，Object_1>
In addition, need to prove, it is example that file storage organization illustrated in fig. 2 is stored in two object sets with each obj ect file, if desired each obj ect file is stored in the plural object set, for example, be stored in three object sets, then a vOSN can be mapped as 3 Objset.
In the embodiment of the invention, be mapped between two Objset on the same vOSN and be regardless of order, but MDS can be according to the master slave relation between definite two Objset of certain strategy.A common object set Objset belongs to certain OSN, and this OSN is an object storage nodes on the network, can measure object storage nodes by processing power, processing power comprises cpu busy percentage, the network bandwidth, hard disk I/O, memory usage, uses amount of capacity etc.MDS is when choosing two Objset on the different OSN and make up, be a unified metric with a plurality of metrics of OSN according to the different switching of its shared weight, this metric is more big to show that then the load of corresponding OSN is more high, so can be with the object set among this OSN as from object set, and the object set on the less relatively OSN of metric be as main object set.
Maximum storage capacity on configurable each vOSN of system is such as 800G.When system initialization, be combined into some vOSN in advance by the Objset that has created on the object storage nodes, in use, when the use amount of finding certain vOSN had reached 80%, then the vOSN that the Objset combination can be made new advances was used for the storage object file.
Corresponding with the embodiment of file memory method of the present invention, the present invention also shows the embodiment of file restoration methods, when breaking down for the file of storing as application file storage means embodiment, these files is recovered.
Participate in Fig. 3, be the first embodiment process flow diagram of file restoration methods of the present invention, this embodiment shows the file rejuvenation when certain object storage nodes breaks down:
Step 301: receive the file recovery request.
Step 302: when comprising the information that object storage nodes breaks down in the file recovery request, search and the object storage nodes corresponding virtual memory node that breaks down.
Step 303: obtain the object storage nodes except the object storage nodes that breaks down that each dummy storage node of finding is shone upon.
Step 304: the object set on the described object storage nodes that breaks down is recovered according to the object set on the object storage nodes of obtaining.
Concrete, the object set on the object storage nodes of obtaining is copied in the object set of object storage nodes of a free time, store the corresponding relation between the object set of object storage nodes of described dummy storage node and described free time.
Below in conjunction with file storage organization shown in Figure 2, describe by the recovery of a concrete application example to the OSN memory node:
The recovery of OSN recovers the Objset on the OSN in fact exactly, and the capacity of Objset can be configured by MDS, so the recovery of obj ect file can be that unit carries out with Objset.After finding certain OSN node collapse, MDS traversal vOSN finds all Objset on this OSN to the mapping table of Objset, finds corresponding backup (antithesis) Objset simultaneously.For each Objset that will recover, MDS creates new Objset at certain idle OSN, and by sending the whole backup of instruction indication OSN copy Objset to idle OSN.OSN reports for the reproduction process of Objset, and MDS gathers current returning to form, and after certain Objset finished recovery, vOSN was to the corresponding relation of new Objset in the updating form.
According to file storage organization shown in Figure 2, suppose to have three files, each file is divided into three obj ect file, and is then corresponding, and file and the corresponding relation between dummy storage node and the file object of the last maintenance of MDS are as shown in table 1 below:
Table 1
| File name | The description of obj ect file |
| File-1 | <vOSN_1，Object_1>，<vOSN_2，Object_1>，<vOSN_3，Object_1> |
| File-2 | <vOSN_1，Object_2>，<vOSN_2，Object_2>，<vOSN_3，Object_2> |
| File-3 | <vOSN_1，Object_3>，<vOSN_2，Object_3>，<vOSN_3，Object_3> |
Dummy storage node and the corresponding relation between the object set of the last maintenance of MDS are as shown in table 2 below:
Table 2
| The vOSN numbering | Two corresponding object sets are described |
| vOSN_1 | <OSN_1，Objset_1>，<OSN_2，Objset_1> |
| vOSN_2 | <OSN_1，Objset_2>，<OSN_3，Objset_1> |
| vOSN_3 | <OSN_2，Objset_2>，<OSN_3，Objset_2> |
Suppose object memory node OSN_3 breaks down, and searches the corresponding vOSN of item that all comprise OSN_3 so from the secondary series of table 2, and the result who searches is vOSN_2 and vOSN_3, and the idle object storage nodes in the supposing the system is OSN_4.
At first recover object set＜OSN_3, Objset_1 〉, according to table 2 as can be known with＜OSN_3, Objset_1 corresponding backup object collection is＜OSN_1 Objset_2.Therefore create an object set＜OSN_4 at OSN_4, Objset_1 〉, and to OSN_1 transmission instruction, indicate it on OSN_4, to write object set, perhaps send instruction to OSN_4, indicate it to go up the reading object collection from OSN_1, according to above-mentioned instruction with the object set＜OSN_1 on the OSN_1, Objset_2〉copy to＜OSN_4 Objset_1 〉.Copy finish after, only need the mapping relations of vOSN_2 in the updating form 2, and need not the fileinfo in the updating form 1, the mapping relations of vOSN_2 are as shown in table 3 below in the table 2 after the renewal:
Table 3
| The vOSN numbering | Two corresponding object sets are described |
| vOSN_2 | <OSN_1，Objset_2>，<OSN_4，Objset_1> |
Secondly, recover object set＜OSN_3, Objset_2 〉, according to table 2 as can be known with＜OSN_3, Objset_2 corresponding backup object collection is＜OSN_2 Objset_2.Therefore create an object set＜OSN_4 at OSN_4, Objset_2 〉, and to OSN_2 transmission instruction, indicate it on OSN_4, to write object set, perhaps send instruction to OSN_4, indicate it to go up the reading object collection from OSN_2, according to above-mentioned instruction with the object set＜OSN_2 on the OSN_2, Objset_2〉copy to＜OSN_4 Objset_2 〉.Copy finish after, only need the mapping relations of vOSN_3 in the updating form 2, and need not the fileinfo in the updating form 1, the mapping relations of vOSN_3 are as shown in table 4 below in the table 2 after the renewal:
Table 4
| The vOSN numbering | Two corresponding object sets are described |
| vOSN_3 | <OSN_2，Objset_2>，<OSN_4，Objset_2> |
Above-mentioned rejuvenation to object storage nodes copies the process of object set exactly.Copying the instruction of object set can be sent by the recovery scheduler program on the MDS, and finished by the synchronization program that copies on the OSN, the synchronization program that copies on the OSN is carried out in the process of duplicate instructions, can copy progress to the MDS report, equally finish copy after, also can copy the information of finishing to MDS report.
Participate in Fig. 4, be the second embodiment process flow diagram of file restoration methods of the present invention, this embodiment shows the file rejuvenation when certain object set of certain object storage nodes breaks down:
Step 401: receive the file recovery request.
Step 402: when comprising the information that the object set on the object storage nodes breaks down in the file recovery request, search the object storage nodes corresponding virtual memory node with the object set place of breaking down.
Step 403: obtain the object storage nodes except the object storage nodes at the object set place of breaking down that the dummy storage node that finds is shone upon.
Step 404: according to the object set on the object storage nodes of obtaining the object set that breaks down is recovered, finish current flow process.
Below in conjunction with file storage organization shown in Figure 2, describe by the recovery of a concrete application example to the object set on the OSN memory node:
When certain OSN finds that record of certain Objset is destroyed on it, perhaps owing to other reason can't Objset of local access the time, then can send to MDS to recovery request by this OSN, trigger recovery for this fault Objset by MDS; Perhaps also can seek another OSN that stores this Objset by OSN and hold consultation, the Objset of fault is recovered out, then restoration result be reported MDS, upgrade corresponding list item information by MDS.After MDS receives recovery request, can do a simple checking earlier, such as the read/write data of fault Objset once, according to the checking result, judge whether to need to send the instruction that recovers to this Objset.Owing to have redundancy relationship between two object sets when the storage object file on two object storage nodes, so rejuvenation is exactly according to this redundancy relationship, obtains another Objset corresponding with fault Objset and copies.
According to file storage organization shown in Figure 2, suppose to have three files, each file is divided into three obj ect file, and is then corresponding, and file and the corresponding relation between dummy storage node and the file object of the last maintenance of MDS allow so as shown in table 1:
Suppose object memory node OSN_3 finds internal object collection＜OSN_3, Objset_2〉break down, then OSN_3 can be to MDS query object collection＜OSN_3, Objset_2〉pair relationhip, MDS checks that table 2 as can be known, with＜OSN_3, Objset_2〉corresponding backup object collection is＜OSN_2 Objset_2 〉; MDS can be according to the load distribution situation, it is less to choose an offered load, the OSN that active volume is bigger, such as choosing object storage nodes OSN_4, MDS returns to OSN_3 to relevant information then, relevant information comprises backup object collection＜OSN_2, Objset_2 〉, for the destination object memory node OSN_4 that recovers object set.Recovering object set＜OSN_3, Objset_2〉process in, at first create an object set＜OSN_4, Objset_2 at OSN_4 〉, send instruction to OSN_2 then, indicate it on OSN_4, to write object set, perhaps send instruction to OSN_4, indicate it to go up the reading object collection from OSN_2, according to above-mentioned instruction with the object set＜OSN_2 on the OSN_2, Objset_2〉copy to＜OSN_4 Objset_2 〉.Copy finish after, only need the mapping relations of vOSN_3 in the updating form 2, and need not the fileinfo in the updating form 1, the mapping relations of vOSN_3 are consistent with table 4 in the table 2 after the renewal, do not repeat them here.
Participate in Fig. 5, be the 3rd embodiment process flow diagram of file restoration methods of the present invention, this embodiment shows the file rejuvenation when the obj ect file in certain object set of certain object storage nodes breaks down:
Step 501: receive the file recovery request.
Step 502: when information that the object in the object set that comprises in the file recovery request on the object storage nodes breaks down, search the object storage nodes corresponding virtual memory node with the object place of breaking down.
Step 503: obtain the object storage nodes except the object storage nodes at the object place of breaking down that the dummy storage node that finds is shone upon.
Step 504: according to the object in the object set of the object storage nodes of obtaining the object that breaks down is recovered, finish current flow process.
Still in conjunction with file storage organization shown in Figure 2, describe by the recovery of a concrete application example to certain object in the object set of OSN memory node below:
When MDS receives the information that certain object in the object set breaks down on the OSN, MDS can confirm further whether the object on this OSN damages, after the affirmation recovery information is hung into the recovery tasks formation, when recovering the formation long enough, after perhaps surpassing Preset Time, by the recovery of MDS startup to all objects.In addition, MDS holds after receiving recovery request, can carry out simple authentication to recovery request, such as a bit of data of this object of read/write, behind the demonstration validation, then sends the instruction that recovers this fault object.
According to file storage organization shown in Figure 2, suppose that client finds object＜OSN_3 when reading file File-1, Objset_2, Object_1〉read error, then client reports MDS with this error message, MDS is from this object＜OSN_3, Objset_2, Object_1〉middle picked at random one piece of data, read checking, if find data mistake appears really, then at object set＜OSN_3 of OSN_3, Objset_2〉in create new object＜OSN_3, Objset_2, Object_9〉(it is 9 that suppose object is concentrated maximum object number), simultaneously at object set＜OSN_3, Objset_2〉backup object collection＜OSN_2, Objset_2〉the new object＜OSN_2 of middle establishment, Objset_2, Object_9 〉, the object number between the object set of two antithesis is consistent.MDS sends object to the OSN_3 node and recovers instruction, and the OSN_3 node copies object＜OSN_2, Objset_2, Object_1 after receiving object recovery instruction〉to＜OSN_3, Objset_2, Object_9 〉; Copy finish after, with the object＜OSN_3 on the OSN_3, Objset_2, Object_1〉be labeled as unavailable, simultaneously with the object＜OSN_2 on the OSN_2, Objset_2, Object_9〉be mapped to＜OSN_2 Objset_2, Object_1.Owing in recovering the process of object, object number is adjusted, so the corresponding relation between file File-1 and dummy storage node and the file object in the updating form 1, as shown in table 5 below:
Table 5
| File name | The description of obj ect file |
| File-1 | <vOSN_1，Object_1>，<vOSN_2，Object_1>，<vOSN_3，Object_9> |
In addition, when the one piece of data in certain object in the object set of certain object storage nodes breaks down, then can return to this failure message the OSN at fault data place, be equivalent to remind the OSN node, the data division of certain object damages.By this OSN node processing, can attempt earlier redistributing one section space at the OSN intra-node, new storage space as this object, fault data place object tools is arrived new space, but do not copy fault data, concentrate from the backup object of this object place object set then fault data is copied, thereby recover the object at fault data and place thereof; Can certainly be according to the description of previous embodiment, directly the object to the fault data place recovers, and does not repeat them here.
Corresponding with the embodiment of file memory method of the present invention and file restoration methods, the present invention also provides the embodiment of document storage system, file recovery system and server.
Referring to Fig. 6, be system embodiment block diagram of the present invention, comprise in this system embodiment block diagram: meta data server 610 and at least two object storage nodes 620, convenient for example, only show three object storage nodes 620 among Fig. 6.
When this system embodiment block diagram is used for the file storage, described meta data server 610, after file to be stored is divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node, create at least two object sets in each object storage nodes, each dummy storage node is mapped to the object set of at least two object storage nodes 620, and described at least two object sets belong to different object storage nodes; Described object storage nodes 620 is for the obj ect file of the dummy storage node correspondence of shining upon by the object set storage.
When this system embodiment block diagram is used for the file recovery, described meta data server 610, be used for receiving the file recovery request, when comprising the information that object storage nodes breaks down in the described file recovery request, search and the described object storage nodes corresponding virtual memory node that breaks down, obtain the object storage nodes 620 except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon, according to the object set on the described object storage nodes of obtaining 620 object set on the described object storage nodes that breaks down 620 is recovered.
Referring to Fig. 7, be an embodiment block diagram of server of the present invention, this server is used for file is stored.
This server comprises: division unit 710, corresponding unit 720, creating unit 730 and map unit 740.
Creating unit 730 is used for creating at least two object sets in each object storage nodes;
Further, this server can also comprise (not shown among Fig. 7): assembled unit, be used for the object set at least two object storage nodes is made up, and the object set of each described combination is corresponding with a dummy storage node.
Further, this server can also comprise (not shown among Fig. 7): monitoring unit, for the memory capacity of each dummy storage node of monitoring; Performance element is used for creating new object set in object storage nodes when described memory capacity surpasses the threshold value that sets in advance, and described new object set is made up.
Referring to Fig. 8, be another embodiment block diagram of server of the present invention, this server is used for file is recovered.
This server comprises: receiving element 810, search unit 820, acquiring unit 830 and recovery unit 840.
Wherein, receiving element 810 is used for receiving the file recovery request;
Acquiring unit 830 is used for obtaining the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon;
Concrete, recovery unit 840 can comprise (not shown among Fig. 8): copied cells, in the object set for the object storage nodes that the object set on the described object storage nodes of obtaining is copied to a free time; Storage unit is for the corresponding relation between the object set of the described dummy storage node of storage and the object storage nodes of described free time.
Further, the described unit 820 of searching also is used for searching the object storage nodes corresponding virtual memory node with described object set place of breaking down when described file recovery request comprises the information that the object set on the object storage nodes breaks down;
Described acquiring unit 830 also is used for obtaining the object storage nodes except the object storage nodes at described object set place of breaking down that the described dummy storage node that finds is shone upon;
Described recovery unit 840 also is used for according to the object set on the described object storage nodes of obtaining the described object set that breaks down being recovered.
Further, the described unit 820 of searching, also be used for when described file recovery request comprises the information that the object in the object set on the object storage nodes breaks down, searching the object storage nodes corresponding virtual memory node with described object place of breaking down;
Described acquiring unit 830 also is used for obtaining the object storage nodes except the object storage nodes at described object place of breaking down that the described dummy storage node that finds is shone upon;
Described recovery unit 840 also is used for according to the object of the object set of the described object storage nodes of obtaining the described object that breaks down being recovered.
As seen through the above description of the embodiments, after in the embodiment of the invention file to be stored being divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node, create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, the obj ect file of the dummy storage node correspondence of shining upon by the storage of object set at least, at least two object sets belong to different object storage nodes; Obj ect file is not directly to be stored on the object storage nodes in the embodiment of the invention that hence one can see that, but by the mapping relations between dummy storage node and the object set, for the storage of obj ect file provide one in the middle of key-course, therefore receive when comprising the recovery request that object storage nodes breaks down, search the object storage nodes corresponding virtual memory node that breaks down with this, obtain the object storage nodes except the object storage nodes that breaks down that each dummy storage node of finding is shone upon, according to the object set on the object storage nodes of obtaining the object set on the object storage nodes that breaks down is recovered; Because when object storage nodes breaks down, only needing to safeguard with this object storage nodes has corresponding mapping relations on the dummy storage node of the relation of depositing, therefore is convenient to the recovery of file, has improved the speed that file recovers.
Those skilled in the art can be well understood to the present invention and can realize by the mode that software adds essential general hardware platform.Based on such understanding, the part that technical scheme of the present invention contributes to prior art in essence in other words can embody with the form of software product, this computer software product can be stored in the storage medium, as ROM/RAM, magnetic disc, CD etc., comprise that at least two instructions are with so that a computer equipment (can be personal computer, server, the perhaps network equipment etc.) carry out the described method of some part of each embodiment of the present invention or embodiment.
Above-described embodiment of the present invention does not constitute the restriction to protection domain of the present invention.Any modification of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., all should be included within protection scope of the present invention.
Claims (15)
1. a file memory method is characterized in that, comprising:
After file to be stored is divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node; Create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, described at least two object sets belong to different object storage nodes, and described at least two object sets are used for storing the obj ect file of the dummy storage node correspondence of shining upon; Preserve the mapping relations table between described dummy storage node, described object storage nodes and described object set;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
2. method according to claim 1 is characterized in that, also comprises:
Object set at least two object storage nodes is made up, and the object set of each described combination is corresponding with a dummy storage node;
Monitor the memory capacity of each dummy storage node;
When described memory capacity surpasses the threshold value that sets in advance, create new object set in object storage nodes, and described new object set is made up.
3. method according to claim 1 is characterized in that, also comprises:
Determine main object storage nodes and from object storage nodes according to the processing power of described at least two object storage nodes.
4. a file restoration methods is characterized in that, is used for the file that file memory method is according to claim 1 stored is recovered, and comprising:
Receive the file recovery request;
When comprising the information that object storage nodes breaks down in the described file recovery request, search the mapping relations table between dummy storage node, object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down;
Obtain the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon;
According to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down is recovered;
Upgrade the mapping relations table between described dummy storage node, object storage nodes and object set.
5. method according to claim 4 is characterized in that, the object set on the object storage nodes that described basis is obtained recovers the object set on the object storage nodes that breaks down, and specifically comprises:
Object set on the described object storage nodes of obtaining is copied in the object set of object storage nodes of a free time;
Store the corresponding relation between the object set of object storage nodes of described dummy storage node and described free time.
6. method according to claim 4 is characterized in that, also comprises:
When comprising the information that the object set on the object storage nodes breaks down in the described file recovery request, search the object storage nodes corresponding virtual memory node with described object set place of breaking down;
Obtain the object storage nodes except the object storage nodes at described object set place of breaking down that the described dummy storage node that finds is shone upon;
According to the object set on the described object storage nodes of obtaining the described object set that breaks down is recovered.
7. method according to claim 4 is characterized in that, also comprises:
When information that the object in the object set that comprises in the described file recovery request on the object storage nodes breaks down, search the object storage nodes corresponding virtual memory node with described object place of breaking down;
Obtain the object storage nodes except the object storage nodes at described object place of breaking down that the described dummy storage node that finds is shone upon;
According to the object in the object set of the described object storage nodes of obtaining the described object that breaks down is recovered.
8. a document storage system is characterized in that, comprising: meta data server and at least two object storage nodes, wherein,
Described meta data server, after file to be stored is divided at least two obj ect file, each obj ect file is corresponded to a dummy storage node, create at least two object sets in each object storage nodes, each dummy storage node is mapped at least two object sets, and described at least two object sets belong to different object storage nodes; Preserve the mapping relations table between described dummy storage node, described object storage nodes and described object set;
Described object storage nodes is for the obj ect file of the dummy storage node correspondence of shining upon by the object set storage;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
9. a file recovery system is characterized in that, the file that is used for document storage system as claimed in claim 8 is stored recovers, and comprising: meta data server and at least two object storage nodes, wherein,
Described meta data server, be used for receiving the file recovery request, when comprising the information that object storage nodes breaks down in the described file recovery request, from dummy storage node, search in the mapping relations table between object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down, obtain the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon, according to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down is recovered, and upgrade described dummy storage node, mapping relations table between object storage nodes and object set.
10. a server is characterized in that, is used for file is stored, and comprising:
Division unit is used for file to be stored is divided at least two obj ect file;
Corresponding unit is used for each obj ect file is corresponded to a dummy storage node;
Creating unit is used for creating at least two object sets in each object storage nodes;
Map unit is used for each dummy storage node is mapped at least two object sets, and described at least two object sets belong to different object storage nodes, and described at least two object sets are used for storing the obj ect file of the dummy storage node correspondence of shining upon; Preserve the mapping relations table between described dummy storage node, described object storage nodes and described object set;
Wherein, described object set is the set of a group objects; At least two described object sets are combined, be mapped as a described dummy storage node.
11. server according to claim 10 is characterized in that, also comprises:
Assembled unit is used for the object set at least two object storage nodes is made up, and the object set of each described combination is corresponding with a dummy storage node;
Monitoring unit is for the memory capacity of each dummy storage node of monitoring;
Performance element is used for creating new object set in object storage nodes when described memory capacity surpasses the threshold value that sets in advance, and described new object set is made up.
12. a server is characterized in that, is used for the file of server stores as described in claim 10 is recovered, and comprising:
Receiving element is used for receiving the file recovery request;
Search the unit, be used for when described file recovery request comprises the information that object storage nodes breaks down, searching the mapping relations table between dummy storage node, object storage nodes and object set and the described object storage nodes corresponding virtual memory node that breaks down;
Acquiring unit is used for obtaining the object storage nodes except the described object storage nodes that breaks down that described each dummy storage node that finds is shone upon;
Recovery unit is used for according to the object set on the described object storage nodes of obtaining the object set on the described object storage nodes that breaks down being recovered, and upgrades the mapping relations table between described dummy storage node, object storage nodes and object set.
13. server according to claim 12 is characterized in that, described recovery unit comprises:
Copied cells is in the object set for the object storage nodes that the object set on the described object storage nodes of obtaining is copied to a free time;
Storage unit is for the corresponding relation between the object set of the described dummy storage node of storage and the object storage nodes of described free time.
14. server according to claim 12 is characterized in that,
The described unit of searching also is used for searching the object storage nodes corresponding virtual memory node with described object set place of breaking down when described file recovery request comprises the information that the object set on the object storage nodes breaks down;
Described acquiring unit also is used for obtaining the object storage nodes except the object storage nodes at described object set place of breaking down that the described dummy storage node that finds is shone upon;
Described recovery unit also is used for according to the object set on the described object storage nodes of obtaining the described object set that breaks down being recovered.
15. server according to claim 12 is characterized in that,
The described unit of searching also is used for searching the object storage nodes corresponding virtual memory node with described object place of breaking down when described file recovery request comprises the information that the object in the object set on the object storage nodes breaks down;
Described acquiring unit also is used for obtaining the object storage nodes except the object storage nodes at described object place of breaking down that the described dummy storage node that finds is shone upon;
Described recovery unit also is used for according to the object of the object set of the described object storage nodes of obtaining the described object that breaks down being recovered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010526834.8A CN101986276B (en) | 2010-10-21 | 2010-10-21 | Methods and systems for storing and recovering files and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010526834.8A CN101986276B (en) | 2010-10-21 | 2010-10-21 | Methods and systems for storing and recovering files and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101986276A CN101986276A (en) | 2011-03-16 |
| CN101986276B true CN101986276B (en) | 2013-07-10 |
Family
ID=43710625
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010526834.8A Active CN101986276B (en) | 2010-10-21 | 2010-10-21 | Methods and systems for storing and recovering files and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101986276B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176903A (en) * | 2019-12-31 | 2020-05-19 | 云汉芯城（上海）互联网科技股份有限公司 | Commodity data recovery method, system, equipment and readable storage medium |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279777B (en) * | 2011-08-18 | 2014-09-03 | 华为数字技术（成都）有限公司 | Method and device for processing data redundancy and distributed storage system |
| CN103064765B (en) * | 2012-12-28 | 2015-12-02 | 华为技术有限公司 | Data reconstruction method, device and cluster storage system |
| CN104202387B (en) * | 2014-08-27 | 2017-11-24 | 华为技术有限公司 | A kind of metadata restoration methods and relevant apparatus |
| US10228745B2 (en) * | 2015-01-29 | 2019-03-12 | Hewlett-Packard Development Company, L.P. | Resuming a system-on-a-chip device |
| CN107943412B (en) * | 2016-10-12 | 2021-10-22 | 阿里巴巴集团控股有限公司 | Method, device and system for splitting partition and deleting data file in partition |
| CN109582213B (en) * | 2017-09-29 | 2020-10-30 | 杭州海康威视系统技术有限公司 | Data reconstruction method and device and data storage system |
| CN110086840B (en) * | 2018-01-26 | 2022-03-11 | 浙江宇视科技有限公司 | Image data storage method, device and computer readable storage medium |
| CN110908590A (en) * | 2018-09-17 | 2020-03-24 | 中国电力科学研究院有限公司 | Distributed storage method and system for transformer substation data |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100530125C (en) * | 2007-08-24 | 2009-08-19 | 成都索贝数码科技股份有限公司 | Safety storage method for data |
| CN101605148A (en) * | 2009-05-21 | 2009-12-16 | 何吴迪 | The framework method of the parallel system of cloud storage |
| CN101673289B (en) * | 2009-10-10 | 2012-08-08 | 成都市华为赛门铁克科技有限公司 | Method and device for constructing distributed file storage framework |
| CN101814045B (en) * | 2010-04-22 | 2011-09-14 | 华中科技大学 | Data organization method for backup services |
-
2010
- 2010-10-21 CN CN201010526834.8A patent/CN101986276B/en active Active
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176903A (en) * | 2019-12-31 | 2020-05-19 | 云汉芯城（上海）互联网科技股份有限公司 | Commodity data recovery method, system, equipment and readable storage medium |
| CN111176903B (en) * | 2019-12-31 | 2020-10-02 | 云汉芯城（上海）互联网科技股份有限公司 | Commodity data recovery method, system, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101986276A (en) | 2011-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101986276B (en) | Methods and systems for storing and recovering files and server | |
| CA2913036C (en) | Index update pipeline | |
| JP5918243B2 (en) | System and method for managing integrity in a distributed database | |
| CN103238140B (en) | Based on the system and method quoting management in the storage system of deduplication for easily extensible | |
| JP5539683B2 (en) | Scalable secondary storage system and method | |
| CN100472463C (en) | Method and apparatus for performing operations on selected data in a storage area | |
| JP5254611B2 (en) | Metadata management for fixed content distributed data storage | |
| US8090792B2 (en) | Method and system for a self managing and scalable grid storage | |
| JP5411250B2 (en) | Data placement according to instructions to redundant data storage system | |
| CN105659213B (en) | Restore without the target drives independent data integrality and redundancy shared in distributed memory system | |
| CN103793425B (en) | Data processing method and device for distributed system | |
| CN103116661B (en) | A kind of data processing method of database | |
| CN101577735B (en) | Method, device and system for taking over fault metadata server | |
| CN102708165B (en) | Document handling method in distributed file system and device | |
| KR101352959B1 (en) | Apparatus and method for database management in active and standby nodes of main memory database management system | |
| JP2016524750A5 (en) | ||
| JP2013544386A5 (en) | ||
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| US11347600B2 (en) | Database transaction log migration | |
| US7702757B2 (en) | Method, apparatus and program storage device for providing control to a networked storage architecture | |
| JP2004334574A (en) | Operation managing program and method of storage, and managing computer | |
| JP2005242403A (en) | Computer system | |
| CN102170460A (en) | Cluster storage system and data storage method thereof | |
| US11403176B2 (en) | Database read cache optimization | |
| CN113377292B (en) | Single machine storage engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 611731 Chengdu high tech Zone, Sichuan, West Park, Qingshui River Applicant after: HUAWEI DIGITAL TECHNOLOGIES (CHENG DU) Co.,Ltd. Address before: 611731 Chengdu high tech Zone, Sichuan, West Park, Qingshui River Applicant before: CHENGDU HUAWEI SYMANTEC TECHNOLOGIES Co.,Ltd. |
|
| COR | Change of bibliographic data |
Free format text: CORRECT: APPLICANT; FROM: CHENGDU HUAWEI SYMANTEC TECHNOLOGY CO., LTD. TO: HUAWEI DIGITAL TECHNOLOGY (CHENGDU) CO., LTD. |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220831 Address after: No. 1899 Xiyuan Avenue, high tech Zone (West District), Chengdu, Sichuan 610041 Patentee after: Chengdu Huawei Technologies Co.,Ltd. Address before: 611731 Qingshui River District, Chengdu hi tech Zone, Sichuan, China Patentee before: HUAWEI DIGITAL TECHNOLOGIES (CHENG DU) Co.,Ltd. |
|
| TR01 | Transfer of patent right |