CN103688257B - Method and device for managing metadata - Google Patents
Method and device for managing metadata Download PDFInfo
- Publication number
- CN103688257B CN103688257B CN201280002998.8A CN201280002998A CN103688257B CN 103688257 B CN103688257 B CN 103688257B CN 201280002998 A CN201280002998 A CN 201280002998A CN 103688257 B CN103688257 B CN 103688257B
- Authority
- CN
- China
- Prior art keywords
- subtree
- mark
- directory
- catalogue
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a method and a device for managing metadata. The method comprises: obtaining to-be-transferred load information; searching directory attribute of a metadata directory, selecting load information matching the to-be-transferred load information in the directory attribute and directory having sub-directory tree marks as a target directory, and determining all the sub-directory trees, which regard the target directory as a root directory, to be to-be-transferred target sub-directory trees; presetting the sub-directory tree marks in the directory attribute of the root directory marked by the sub-directory tree marks; and integrally transferring the target sub-directory trees. The method and the device allow metadata management efficiency of a file system to be improved.
Description
Technical field
The present invention relates to memory technology, more particularly to a kind of metadata management method and device.
Background technology
Cluster file system is the system that the file of the multiple stage machine included to cluster is managed collectively, this document system
The memory space unified to the offer of multiple stage machine of uniting is referred to as name space, each machine that the name space is used in storage cluster
In file storage path;For example, the storage path of file a is C/aa/c/a (C/aa/c therein is properly termed as catalogue);
Metadata includes the described information for representing file storage path.Name space is divided into many parts, and per part is distributed to collection
A machine in group, for storing the metadata corresponded in machine;Assume the space storage under the C catalogues in certain machine
C/aa/c/a can be then referred to as a subtree by two file paths, respectively C/aa/c/a and C/bb/d, and C/bb/d is also referred to as
For a subtree, it is actually also metadata that aa/c/a is referred to as a subtree, i.e. subtree, simply according in metadata
The vivid appellation of the hierarchical linkage relation setting between catalogue or file.
In the running of cluster, file can be accessed by the application of user, and the visit capacity of file can be with load table
Show, the load of the bigger machine of file access amount of certain machine is higher.In order that the load balancing of each machine in cluster,
Prior art can be in the heavier loads of certain machine, will be (actual on a part of load migration of the machine to another machine
It is the migration of file);Accordingly, the metadata of the machine also will be updated with the movement of load, will be right with migration load
The subtree answered also moves to another machine, i.e. load migration its corresponding metadata and also can migrate, and the distribution of metadata occurs
Change.
But, in the load migration and subtree migration carried out for load balancing, metadata can be caused in the cluster
Distribution it is more scattered, it is possible to the situation of appearance is, it is assumed that certain file is moved to another by user from a catalogue
When under catalogue, and the two catalogues are located at respectively on different machines, cause the distribution management to metadata to carry out across machine
Device operate, will metadata be sent to another machine from a machine, so as to cause file system to carry out file behaviour in user
Less efficient, the access performance for affecting cluster to apply user of the meta-data distribution management accordingly carried out when making.
The content of the invention
The present invention provides a kind of metadata management method and device, to improve the metadata management efficiency of file system.
A kind of first aspect, there is provided metadata management method, including:
Obtain the load information of migration;
The directory attribute of the catalogue in metadata is searched, selects directory attribute to include and the load information by migration
The load information for matching and the catalogue with subtree mark are target directory, it is determined that the institute with the target directory as root
There is subtree as the target subtree that will be migrated;The subtree mark is set in advance in the root identified by the subtree mark
Directory attribute in;
By the target subtree bulk migration.
With reference in a first aspect, in the first possible implementation, the subtree of the root of the subtree mark mark
Include the corresponding metadata of minimum unit of Operational Visit opereating specification.
With reference in a first aspect, in second possible implementation, the subtree of the root of the subtree mark mark
In be included in the metadata that visiting frequency in preset time period reaches predetermined threshold value.
With reference in a first aspect, in the third possible implementation, the directory attribute of the catalogue in metadata is searched
Before, also include:In the directory attribute of root has at least two subtrees of subtree mark, select to become with identical load
Multiple subtrees of change trend as a concurrent group, and by the directory attribute of the root of each subtree in the concurrently group
Concurrent group mark is set;The selection directory attribute include with the load information that the load information of migration matches and
With and subtree mark catalogue be target directory, including:Select not only to include the load information in directory attribute and have
Subtree mark, also including the concurrent group mark catalogue as the target directory.
With reference to any one in the first possible implementation to the third possible implementation of first aspect,
In the 4th kind of possible implementation, in the selection directory attribute load information and subtree mark are not only included,
Also including the concurrent group mark catalogue as the target directory, including:From the concurrently group most including subtree quantity,
Select not only to include the load information and subtree mark in directory attribute, also make including the catalogue of the concurrent group mark
For the target directory.
With reference to the third possible implementation of first aspect, in the 5th kind of possible implementation, the catalogue
Attribute is the attribute that receive user is arranged by application programming interface API, and the attribute is including the subtree mark and simultaneously
Send out group mark.
A kind of second aspect, there is provided meta data management device, including:
Load determining unit, for obtaining the load information of migration;
Sub-tree search unit, for searching metadata in catalogue directory attribute, select directory attribute include and institute
State load information that the load information of migration matches and the catalogue with subtree mark is target directory, it is determined that with the mesh
Heading record is all subtrees of root as the target subtree that will be migrated;The subtree mark is set in advance in by the subtree
In the directory attribute of the root that mark is identified;
Subtree migration units, for by the target subtree bulk migration.
With reference to second aspect, in the first possible implementation, the subtree of the root of the subtree mark mark
Include the corresponding metadata of minimum unit of Operational Visit opereating specification.
With reference to second aspect, in second possible implementation, the subtree of the root of the subtree mark mark
In be included in the metadata that visiting frequency in preset time period reaches predetermined threshold value.
With reference to second aspect, in the third possible implementation, also include:Attribute setting unit, for described
Sub-tree search unit is searched before the directory attribute of the catalogue in metadata, has subtree mark in the directory attribute of root
In at least two subtrees, select the multiple subtrees with identical load variation tendency as a concurrent group, and will it is described concurrently
Concurrent group mark is set in the directory attribute of the root of each subtree in group;The sub-tree search unit, specifically for looking into
The directory attribute of the catalogue looked in metadata, selects not only to include the load information in directory attribute and with subtree mark,
Catalogue also including the concurrent group mark determines all sons with the target directory as root as the target directory
Set as the target subtree that will be migrated;The subtree mark is set in advance in the mesh of the root identified by the subtree mark
In record attribute.
With reference to any one in the first possible implementation to the third possible implementation of second aspect,
In the 4th kind of possible implementation, the sub-tree search unit, specifically for from including the most concurrently group of subtree quantity
In, select not only to include the load information and subtree mark in directory attribute, also including the catalogue of the concurrent group mark
As the target directory.
With reference to the third possible implementation of second aspect, in the 5th kind of possible implementation, also include:Category
Property acquiring unit, for the attribute that receive user is arranged by application programming interface API, the attribute include the subtree
Mark and concurrent group mark.
The third aspect, there is provided a kind of calculate node for metadata management, the calculate node includes:It is processor, logical
Letter interface, memory and bus;Wherein, the processor, communication interface and memory complete mutual by the bus
Communication;
The communication interface, the calculate node for the metadata management receives program;
The processor, for configuration processor;
The memory, for depositing program;
The computing device described program causes the calculate node of the metadata management to perform any of the above-described method.
The metadata management method of present invention offer and having the technical effect that for device:By arranging subtree in directory attribute
Mark, and by all subtrees using the catalogue with subtree mark as root as the target subtree bulk migration that will be migrated,
Relative to the scattered migration pattern of metadata of the prior art, can effectively reduce the metadata operation across MDS, save across
During MDS between difference MDS information transfer time, meta-data distribution management efficiency be improved.
Description of the drawings
Fig. 1 is the meta-data distribution figure of the file system of metadata management method embodiment application of the present invention;
Fig. 2 is the schematic flow sheet of the embodiment of metadata management method of the present invention;
Fig. 3 is the schematic flow sheet of another embodiment of metadata management method of the present invention;
Fig. 4 is the file system metadata distribution map of another embodiment of metadata management method of the present invention;
Fig. 5 is the MDS clusters overall performance contrast chart one of metadata management method embodiment of the present invention;
Fig. 6 is the MDS clusters overall performance contrast chart two of metadata management method embodiment of the present invention;
Fig. 7 is the MDS cluster load distribution charts before the improvement of metadata management method embodiment of the present invention;
Fig. 8 is the MDS cluster load distribution charts after the improvement of metadata management method embodiment of the present invention;
Fig. 9 is the structural representation of the embodiment of meta data management device of the present invention;
Figure 10 is the structural representation of another embodiment of meta data management device of the present invention;
Figure 11 is the structural representation of the calculate node embodiment of metadata management of the present invention.
Specific embodiment
In order that the metadata management method of the embodiment of the present invention is easier to understand, first with reference to Fig. 1, to the present invention
The basic concepts being related in embodiment are illustrated, and Fig. 1 is the text of metadata management method embodiment application of the present invention
The meta-data distribution figure of part system.The metadata management of the present embodiment is actually managed to the metadata in file system
Reason, specifically to the management of meta-data distribution;Described metadata is the management information with regard to catalogue or file etc., for example,
Title, attribute, hierarchical relationship of catalogue or file etc..
Referring to Fig. 1, with meta data server (Metadata server, abbreviation:MDS) as a example by cluster, in the cluster,
Including multiple MDS, each MDS is responsible for the metadata of a machine;Described metadata be shown in Fig. 1 by mutual
The tree-shaped information of each square frame composition of connection, for example, k1, bucket_1 etc..Three memory spaces have been outlined with a dotted line in Fig. 1
Scope, be belonging respectively to MDS1, MDS2 and MDS3, for example, the metadata being stored in MDS1 in dotted line frame p1, dotted line frame
The metadata being stored in MDS2 in p2, the metadata being stored in MDS3 in dotted line frame p3.Not in dotted line frame model
Metadata in enclosing is the metadata on other MDS.
The metadata of the MDS clusters is that the metadata of the tree is related to the tree storage shown in Fig. 1
The following concept, wherein it is desired to first number that explanation, catalogue, file, subtree, node for being previously mentioned as follows etc. each mean
According to, embodiment of the present invention discussion be metadata distributed architecture, for example, the file k_e.avi in the Fig. 1 being previously mentioned is actual
It is the metadata of file k_e.avi;Additionally, the load being previously mentioned in the embodiment of the present invention also refer to for metadata access
Load, but the renewal of the metadata load is also by caused by the actual access for file or catalogue.
Catalogue:Kobe, james, bucket_1 for example shown in Fig. 1 etc.;
File:K_e.avi, k_d.avi for example shown in Fig. 1 etc., file under catalogue, such as, and k_e.avi
It is to be located under catalogue k3, the upper level catalogue of catalogue k3 is k2, and the upper level catalogue of catalogue k2 is k1_1, the like, should
Catalogue "/" at the top of tree is total root;
Above-mentioned file and catalogue and its between hierarchical relationship (i.e. file is located under certain catalogue) constitute file
Store path;For example, the store path of file k_b.avi is /kobe/bucket_1/k1/k_b.avi.Again for example, in Fig. 1
File is not also placed under catalogue bucket_2.
Subtree:If the catalogue "/" at the top of the tree in Fig. 1 likened into into the root of big tree, the big tree it is any
One branch can be referred to as a subtree;
For example, all metadata (including all of catalogue and file) of catalogue kobe and its subordinate are collectively referred to as one
All metadata of subtree, catalogue k1 and its subordinate are collectively referred to as a subtree (i.e. k1-k_b.avi), catalogue k2 and its under
All metadata of category are collectively referred to as a subtree (subtree includes k2, k3, k_e.avi, k_c.avi).Can see,
Subtree is i.e. with certain catalogue as root, including the catalogue and all branches for being connected under the catalogue, and this is collectively referred to as one
Subtree.
It is to be described with the migration of subtree, because the migration of metadata is in the migration of follow-up described metadata
In units of subtree.
Node:Each square frame in Fig. 1 is referred to as into a node, it can also be file that the node can be catalogue;Example
Such as, k_e.avi, bucket_1, kobe etc. are a nodes;
The attribute of node and load:The attribute of node for example includes creation time, load value etc., each catalogue or file
There will be individual creation time, the creation time is exactly an attribute of node;Each node that load value may refer in Fig. 1 is left
The numeral that superscript shows, for example, the 31 of the bucket_1 upper left corners, 71 etc. of the kobe upper left corners.The load value of node illustrates this
The present load of node, is the parameter of the visit capacity for characterizing the file under node, and the visit capacity of usual file gets over heavy load also just
It is higher, thus the load that the MDS that the load value of node is actually also the node to be located is born, certainly, the load of whole MDS
It is the summation of the load value of all nodes disposed thereon.
Such as, the load value of bucket_1 be the load value of 31, kobe be 71;The load value of certain catalogue is to be connected to this
The summation of the load value of the next stage node (catalogue or file) under catalogue, such as, from figure 1 it will be seen that file k_e.avi
Be under catalogue k3, the load value of this document k_e.avi be 2 (load value be according to file access amount or access frequency etc. according to
The calculated characterization value of certain algorithm), the load value of catalogue k3 is identical with the load value of this document k_e.avi;And mesh
The load value of record k2 is the summation of the load value of the file k_e.avi and file k_c.avi of its subordinate, the load value=k3's of k2
The load value of load value+k_c.avi.
The migration of subtree:Fig. 1 includes many subtrees, and this multiple subtree is to be belonging respectively to different MDS;Than
Such as, subtree bucket_1/k1/k_b.avi is located on MDS1, namely represents file k_b.avi, catalogue k1 and catalogue bucket_1
Metadata be by MDS1 manage and provide service, the metadata to this document k_b.avi, catalogue k1 and catalogue bucket_1
Visit capacity it is bigger by cause MDS1 load it is larger;And subtree k2_1/k_d.avi is stored on MDS2, namely represent
The metadata of file k_d.avi and catalogue k2_1 is stored in MDS2, to the visit capacity of this document k_d.avi or catalogue k2_1 compared with
Senior general causes the load of MDS2 larger.The load of MDS refers to the load of the metadata on the MDS, and the load of metadata
It is to be caused by the access of the metadata corresponding actual catalogue or file.It is exemplified below:Hypothesis is entered to file k_b.avi
Go actual access, then the load value of the metadata of the corresponding this document k_b.avi being stored on MDS1 will increase, and
And, after the load value of the metadata of file k_b.avi increases, the load value of its higher level's catalogue also can increase therewith, such as catalogue
The metadata load value of bucket_1/k1 increases;Or, it is assumed that catalogue k2_1 is accessed, then it is corresponding to be stored in
The load value of the metadata of catalogue k2_1 on MDS2 will increase.
In order to keep load balancing between this multiple MDS, it will usually when the load of current MDS is larger, one is migrated
Divide and load to another MDS, actually by the metadata migration of some files and catalogue to another MDS, thus by another
One MDS reduces the load of current MDS to the file of the migration and the access service of catalogue.During load migration, file and mesh
The storage location of the metadata of record changes, such as be transferred to MDS3 by MDS2.It should be noted that the migration of metadata
It is with subtree as unit, such as subtree k2_1/k_d.avi can only to be migrated.The migration of subtree is by the storage location of subtree
Change, subtree k2_1/k_d.avi, but the subtree are framed in whole tree equivalent to by dotted line frame p3 in Fig. 1
Annexation be not changed in, such as, subtree k2_1/k_d.avi is remained and is connected under catalogue k1_1.
On the basis of the above description, the metadata management method of the embodiment of the present invention is described below in detail:
Embodiment one
Fig. 2 is the schematic flow sheet of the embodiment of metadata management method of the present invention, as shown in Fig. 2 the method can be wrapped
Include:
201st, obtain the load information of migration;
Wherein, described load information refers to load value, than as shown in Figure 1 in each node upper left corner sign
Numeral, for example, the 31 of the bucket_1 upper left corners.
What the present embodiment was obtained is, by the load information of migration, to be exemplified below:Cluster includes three MDS, each MDS
Load value in corresponding metadata can be updated according to access informations such as the visit capacities of its own responsible file and catalogue, and
And the difference between the total load value that can be carried with periodic detection itself and other MDS load values, when the difference beyond
Certain threshold value then shows that the load distribution between each MDS is unbalanced, then the MDS of heavier loads is by starting load equilibrium treatment,
By on a part of load migration of itself to other MDS.The migration of the actually load is certain subtree in metadata
Migration, is that the corresponding access of the subtree is moved to into another MDS, and described access is serviced by another MDS, corresponding negative
The load that the heavier MDS of load is carried is reduced.
In the present embodiment, what such as certain MDS got is 7 load values by the load information of migration, that is, need 7
Load value moves to other MDS.
202nd, the directory attribute of the catalogue in metadata, the load information for selecting directory attribute to include and will migrate are searched
The load information for matching and the catalogue with subtree mark are target directory, it is determined that the institute with the target directory as root
There is subtree as the target subtree that will be migrated;Subtree mark is set in advance in the catalogue category of the root identified by subtree mark
In property;
Wherein, after determining the load information that will be migrated in 201, by the directory attribute for searching the catalogue in metadata,
Directory attribute is selected to include with the load information that the load information of migration matches and with the catalogue of subtree mark
For target directory;Load information in the directory attribute of the target directory represents the maximum son with the target directory as root
Set corresponding load value.It should be noted that described matching refers to the corresponding load value of subtree believing with described load
Breath is not necessarily equal, can be close, and the close scope of specific numerical value can be by the fuzzy matching algorithm of conventional load balancing
It is determined that, the embodiment of the present invention is no longer described in detail.
It is exemplified below:May refer to Fig. 1, it is assumed that be that MDS2 migrates out 7 load values of itself, in fig. 2 may be used
To see, the load of subtree k2_1/k_d.avi is exactly that (load value of catalogue k2_1 upper left corner mark is 7 to 7 load values, and this is born
Load value is also one of attribute of catalogue k2_1, is properly termed as load attribute);Now, the corresponding load value of subtree with it is described
Load information be equal.And if MDS2 migrates out 8 load values of itself, and assume that it is not present with complete
The subtree of complete equal load value, it is also possible to migrate out subtree k2_1/k_d.avi that above-mentioned load value is 7,8 and 7 numbers
Value is close, it is believed that the corresponding load value of subtree matches with described load information.
It should be noted that the subtree of the migration selected in the present embodiment, not only load value and the load information phase
Matching, and the root of the subtree be catalogue k2_1 attribute also include subtree flag property.
Front illustrated the concept of subtree already in connection with Fig. 1, and subtree is i.e. with certain catalogue as root, including the catalogue and company
The all branches being connected under the catalogue, this is collectively referred to as a subtree;Then root should be referred to herein as the catalogue of root.Than
Such as, the root of subtree k2_1/k_d.avi is catalogue k2_1;Including k2/k3/k_e.avi and k2/k_c.avi Liang Ge branches
Subtree, its root is catalogue k2.The selected subtree to be migrated of the present embodiment, its root must have subtree mark.
Described subtree mark is intended to indicate that all subtrees with the target directory as root are only capable of with overall side
Formula is migrated, and all subtrees of the present embodiment determination using the target directory as root are used as the target subtree that will be migrated.
Optionally, the characteristics of target subtree with the target directory as root of the present embodiment can have following:
The metadata that the target subtree includes is the minimum unit of correspondence Operational Visit opereating specification, and for example, user is a certain in execution
During business, used an application (certain application program that i.e. user uses), the application to the corresponding catalogue of the subtree or
The access of file, is essentially all the access inside the subtree, and target subtree is included using the corresponding all units of access operation
Data (all metadata can be equivalent to the minimum unit of Operational Visit opereating specification).Or, user is performing a certain industry
During business, two changes for being related to a part of metadata using A and B, A, B have been used to be related to another part metadata
Change, this two parts metadata has occurs simultaneously, then the metadata that the metadata and B that can be related to A is related to is used as one
Overall (entirety can be equivalent to the minimum unit of Operational Visit opereating specification), the entirety is entirely included in target
In tree.Or, without common factor, the then unit that will can be related to using A between the metadata that above-mentioned two application A and B are related to
Data are separately as an entirety (entirety can be equivalent to the minimum unit of Operational Visit opereating specification), the target subtree
Include the entirety completely.Simply illustrate above, this can be not limited to flexible in being embodied as.
Can be with reference to the meaning that the metadata that a kind of optional described target subtree includes is illustrated as a example by the subtree in Fig. 1
Think:Such as, the subtree including k2/k3/k_e.avi and k2/k_c.avi Liang Ge branches in Fig. 1, by the root k2 of the subtree
Subtree flag property is set, then the subtree is an entirety;User for example includes using the access to the subtree:User is by file
K_e.avi is moved under catalogue k2 from catalogue k3, or, file k_c.avi is moved to into catalogue k3 from catalogue k2, or,
Create file etc. again under catalogue k3.Can see, these operations are all the operations inside the subtree, are not related to
Other subtrees outside subtree, described subtree has been included using the corresponding metadata of operation is accessed, for example, using access
Operation is k_e.avi to be moved under catalogue k2 from catalogue k3, and its metadata being related to is k_e.avi, k3, k2, these
Metadata is all inside the subtree;Catalogue k3 is set to subtree mark by hypothesis, and corresponding subtree k3/k_e.avi does not actually have
Have and access metadata k2 that operation is related to including application, that is, only included a part of metadata, such case does not just meet this
The situation of the metadata that the target subtree described in embodiment includes.
Optionally, the target subtree with the target directory as root of the present embodiment can also have following spy
Point:The metadata that the target subtree includes is correspondence experience Transport;I.e., it is possible to the metadata in each MDS
The characteristics of change, carries out the setting of subtree mark, for example, if it find that certain a part of metadata Jing is often same within a period of time
When have access to, and the visiting frequency of this part metadata has reached predetermined threshold value in preset time period, for example, the default threshold
Value is 50 times, and the visiting frequency of metadata has reached 50 times in preset time period, then can determine including these metadata
Subtree it is overall as target subtree.
Described means the corresponding subtree bulk migration of the catalogue for being provided with subtree mark, with including k2/k3/k_
As a example by the subtree of e.avi and k2/k_c.avi Liang Ge branches, the effect to subtree flag property is illustrated:Assume the subtree
Root k2 is not provided with subtree flag property, then the branch of the subtree also can be in fact segmentation, such as, by its branch
K3/k_e.avi individually migrates (branch is also in fact a subtree), that is, migrated out 2 load values;But, if by root
Catalogue k2 arranges subtree flag property, then show the described son including k2/k3/k_e.avi and k2/k_c.avi Liang Ge branches
Tree, it is impossible to be divided again, but can only bulk migration, it is impossible to a part therein is only migrated again.
In being embodied as, when searching the subtree to be migrated, the present embodiment be to search metadata in catalogue performing
, if the attribute of certain catalogue includes subtree mark and the load attribute of the catalogue matches with the load information,
Show, the load value of the subtree with the catalogue as root seeks to the load value for migrating, and the subtree can only be with overall side
Formula is migrated, here it is the subtree to be migrated, the root of the subtree for finding is referred to as target directory by the present embodiment.
203rd, by the target subtree bulk migration.
The metadata management method of the present embodiment, is realized in improving the metadata management efficiency of file system:It is logical
Cross in directory attribute setting subtree mark, and by all subtrees using the catalogue with subtree mark as root as will move
The target subtree bulk migration of shifting, relative to the scattered migration pattern of metadata of the prior art, because target subtree includes
This part metadata migrate in an integral manner all the time, necessarily can effectively reduce the metadata operation across MDS, such as when some
When the metadata reality for being related to change is all the metadata inside the target subtree, just variation is only inside the target subtree
Can, will not relate to the operation across MDS, during so as to saving across MDS between difference MDS information transfer time, metadata point
The efficiency of cloth management is improved.
I.e., on the one hand, the distribution of metadata is that, by MDS self-managements, MDS is responsible for carrying out load in heavier loads moving
Move;When metadata is distributed between each MDS, the subtree being located with the root of subtree flag attribute can be integrally placed to
In same MDS, without each several part in such subtree is individually positioned in into different MDS, therefore, in metadata
During distribution, it is possible to according to the subtree flag attribute, by the subtree entirety with the catalogue with subtree flag attribute as root
Positioned at same MDS.On the other hand, in the process of load balancing, even if to migrate the subtree including target directory, be also by
The subtree bulk migration, the part migration without only splitting the subtree, so ensures that the entirety of the subtree always
In same MDS, without being located at different MDS respectively.
Illustrate the effect so that the metadata that target subtree includes is all metadata that certain application is related to as an example again
Embody:The access operation for assuming certain user's application is file k_e.avi to be moved under catalogue k2_1 from catalogue k3, accordingly,
Because the store path of this document changes, (can be understood as metadata is exactly the store path for representing file to corresponding metadata
) respective change is also wanted, it is exactly the k_e.avi in the tree in Fig. 1 to be connected to into catalogue k3 and is changed to specific to Fig. 1
It is connected to catalogue k2_1;If subtree k2_1/k_d.avi is located on MDS3, and subtree k3/k_e.avi is located on MDS2, then
When meta-data distribution is changed, the metadata of file k_e.avi will be sent to MDS3 by MDS2 by across MDS operation, by MDS3
To store metadata i.e. subtree k2_1/k_e.avi after updating, just by MDS3 offer texts during the access of subsequent file k_e.avi
The service of part store path, the access reality of this document just becomes the load of MDS3.If subtree k2_1/k_d.avi and son
Tree k3/k_e.avi is all located at MDS2, then above-mentioned meta-data distribution is changed just in MDS built-in functions, is eliminated across MDS
The time of information transfer between Shi Butong MDS, the efficiency that meta-data distribution updates is improved.
The following examples two and embodiment three, will describe the present invention in detail by two optional specific examples
The enforcement of the meta-data distribution management method of embodiment.
Embodiment two
Fig. 3 is the schematic flow sheet of another embodiment of metadata management method of the present invention, and the present embodiment is mainly described how
Certain load value is migrated out from certain MDS;Fig. 4 is the file system unit of another embodiment of metadata management method of the present invention
Data profile, the Fig. 4 is illustrated that the meta-data distribution structure on the one of MDS in cluster.With reference to Fig. 3 and Fig. 4 institutes
Show, the method includes:
301st, subtree flag property is set for catalogue;
Wherein, it is to include multiple MDS in cluster, is managed collectively the cluster file system of metadata of the cluster in Jiang Ji
Group's metadata can first arrange the attribute of the metadata, described setting metadata attributes before multiple MDS are distributed
Work be by the cluster metadata control module perform;The metadata control module is responsible for metadata and sets a property and incite somebody to action
Multiple MDS in meta-data distribution to cluster.Unit after initial meta-data distribution is completed, in follow-up cluster running
Data distribution processes such as load balancing etc., is just performed by each MDS oneself, and each MDS manages respectively the metadata of oneself storage, and
The distribution of metadata according to load management.
In the present embodiment, described metadata control module is that the catalogue in metadata is provided with subtree flag property;Institute
The setting principle of the subtree flag property stated is such:Application is accessed involved metadata as an entirety, will
The root of the overall corresponding subtree arranges subtree flag property, with ensure the access applied it is corresponding be with the target mesh
Record the metadata evolution inside the subtree for root.
It is exemplified below:When certain application program is enabled, the corresponding access operation of the application program is included file user
K_e.avi is moved under catalogue k2_1 from catalogue k3, or file k_c.avi is moved to into catalogue k3 from catalogue k2, or
It is new under catalogue k3 to create file etc..Can be obtained according to the accessing characteristic of the application program, if will including k1_1,
The metadata of k2_1, k2, k3, k_e.avi, k_c.avi and k_d.avi is used as an entirety, then can be so that above-mentioned should
Access it is corresponding be the whole interior metadata evolution;Generally one subtree, by the overall root k1_1
Subtree flag property is set, and described subtree mark is used to represent that the corresponding subtrees of root k1_1 are only capable of moving in an integral manner
Move.
Can also be learnt by above-mentioned, actually subtree flag property is arranged according to the accessing characteristic of application, based on this,
The present embodiment is provided with the following two kinds attribute set-up mode:
A kind of mode is to provide gives user application DLL (Application Programming
Interface, referred to as:API), by user directly the attribute of catalogue is configured according to the accessing characteristic of the application of user;
If i.e. using with above-mentioned accessing characteristic, the attribute of catalogue just can be set to subtree mark by user accordingly.
Another way is to perform the setting of directory attribute automatically by above-mentioned metadata control module;For example, Ke Yiyou
User arranges an attribute Provisioning Policy for the metadata control module, and the strategy is for example above-mentioned " access correspondence application
Metadata evolution be subtree inside change subtree be set to an entirety " etc.;After good strategy is set, metadata control mould
The application that block just can occur in the running of cluster accesses operation, detects that the application accesses the accessing characteristic of operation, such as
Fruit meets above-mentioned strategy, then metadata control module can be arranged the attribute of the root of the overall subtree according to the strategy
For subtree mark.
In being embodied as, referring to Fig. 4, subsequently search which catalogue has subtree flag property, can be arranged for convenience
One subtree queue can be each MDS in its internal subtree queue for each setting up correspondence itself storage metadata, the subtree team
The catalogue of all configured subtree flag properties is included in row;Or, MDS arranges a son for including its all metadata
Tree queue, and a pointer is inserted in subtree queue, the pointer points to the catalogue for being provided with subtree flag property.For example, this reality
In applying example, catalogue k1_1, j1 and j2 are provided with subtree flag property.
Metadata control module also arranges other attributes for metadata, for example, the creation time of catalogue or file, load
Value etc.;After setting a property for cluster metadata according to above-mentioned rule, metadata control module by meta-data distribution in each MDS,
Wherein, the corresponding subtree of root for being provided with above-mentioned subtree flag property will be integrally dispensed in same MDS.
It should be noted that the present embodiment is only illustrated as a example by arranging subtree flag attribute when initial;But tool
Body is not limited thereto in implementing, and the setting of the metadata attributes for example includes the setting of the subtree flag attribute, during its setting
Between without limit, it is also possible in MDS system operations at any time according to using the characteristics of the subtree flag attribute is set.When in system
When the subtree flag attribute is provided with operation, if the subtree with the catalogue of the subtree flag attribute as root is integrally point
On different MDS, then system can be according to the subtree flag attribute, by the subtree bulk migration to same MDS for cloth.It is being
The mode that subtree flag attribute is arranged in system operation is identical with aforesaid mode, such as API modes or automatically setting.
302nd, obtain the load information of migration;
Wherein, in the running of cluster, the distribution of the metadata of storage is managed by each MDS oneself, and performs each MDS
Between load balance process.
By taking the MDS1 shown in Fig. 4 as an example, the MDS1 can be according to the negative of the metadata for accessing renewal oneself storage of application
Value attribute is carried, such as, certain application that user enables currently repeatedly have accessed file k_b.avi, and MDS1 can be accessed for this document
Routing Service is provided, i.e., is applied according to the file path guide for being stored and is had access to file k_b.avi, and accordingly, MDS1
The load value of the metadata involved by the access can be updated according to above-mentioned file access, specifically for example update text
The load value of part k_b.avi, by 10 12 are increased to, accordingly, higher level's catalogue of this document k_b.avi, such as catalogue k1,
Bucket_1, kobe and/can all carry out load value renewal, increase by two load values.
Outside load value is updated, the load difference that MDS1 can also be between each MDS of periodic detection;Including multiple MDS's
In cluster, it is in communication with each other between each MDS, its respective load information etc. can be interacted, so, MDS1 is to get
The load information of other MDS, and the load information of itself is compared with the load information of other MDS, judge whether to reach
The trigger condition for performing load balancing is arrived.For example, if the load between the load of MDS1 and other certain MDS can be set
When difference reaches 20 load values, load balancing is carried out with regard to triggering, to balance the load distribution between each MDS.In the present embodiment,
Assume that MDS1, through detection, determines that own load is overweight, need for 8 load values to move to other MDS, that is, what is got will move
The load information of shifting is 8 load values.
303rd, search whether there is target directory, attribute includes subtree mark and load attribute matches with load information;
Specifically, this step is to look in metadata with the presence or absence of target directory, and the target directory attribute includes subtree
Indicate and the load attribute of the target directory matches with the load information.
Wherein, still by taking the MDS1 in Fig. 4 as an example, MDS1 by it is preferentially searched it is determined that after by the load information of migration, deposit
The catalogue of subtree flag property is provided with the metadata of storage, and whether judge with the load attribute of the catalogue (i.e. load value)
The load information of migration is matched with described.In the present embodiment, preferential lookup had into the load equal with load information
The subtree of value.
It is exemplified below:MDS1 will be searched preferentially in catalogue k1_1, j1 and j2, because these three catalogues are all provided with subtree
Flag property;Judge whether to have load attribute in these three catalogues with the described catalogue that the load information of migration matches.
Through judging, the load value of these three catalogues is not 8, therefore, without suitable catalogue.
In this step, if through searching, there is described target directory, i.e. attribute is its load attribute of subtree mark
It is identical with the load information, then 305 are continued executing with, by the subtree bulk migration with the target directory as root;Otherwise, such as
Through searching, there is no described target directory in discovery to fruit, then continue executing with 304.
304th, the suitable catalogue of searching is begun stepping through from the dynamic subtree root of this MDS, into recurrence;
Wherein, the dynamic subtree root of this described MDS, that is, refer to catalogue kobe, catalogue james in MDS1, from Fig. 4
In it can also be seen that the two catalogues are the initial roots of all metadata in MDS1, other catalogues or file are all
Extend from the two contents segmentations, it is possible to referred to as subtree root;Also, because the subtree of the present embodiment all can be segmentation
, such as, it is two subdirectories etc. by certain contents segmentation, so being referred to as dynamic subtree root.
Described traversal finds suitable catalogue, into recurrence, refers to along the TOC level shown in Fig. 4, step by step
Look for whether the catalogue of load value matched with load information;When running into during searching with subtree flag property
During catalogue, then stop and return upper level catalogue.
It is exemplified below:When being 8 load values by the load information of migration, MDS1 starts to seek downwards step by step from catalogue kobe
Look for, and judge whether the load value of catalogue is 8, for example, in the following order, kobe --- bucket_1 --- k1 --- k_
B.avi, kobe --- bucket_1 --- k1_1, etc., wherein, actually in described kobe --- bucket_1 ---
K1 --- in k_b.avi paths, the load value for just having been found that catalogue k1 is 8, then continue executing with 305.
Assume that MDS1 first searches for kobe --- bucket_1 --- k1_1 paths, then when finding catalogue k1_1, finding should
Catalogue k1_1 has subtree flag property, then will not be further continued for searching the subdirectory or file (such as mesh under catalogue k1_1
Record k2, file k_e.avi etc.) because the subtree of the root with subtree flag property is an entirety, it is not necessary that followed by
Continue the internal searching in the overall subtree.Then MDS1 will stop at catalogue k1_1, and it is bucket_1 to return upper level catalogue,
Again from the bucket_1 along the path searching of bucket_1 --- k1 --- k_b.avi.
305th, the subtree chosen and the load information of migration matches, is migrated.
Wherein, as above, the load value of catalogue k1 is 8, it is determined that subtree k1-k_b.avi is the subtree of migration.
Can see from above-mentioned flow process, when certain load value is migrated out from certain MDS, preferential lookup be set
The catalogue of subtree flag property is put;Also, the subtree in the subtree of the root with subtree flag property is will not be by
Select, i.e., the subtree of the root with subtree flag property is an entirety, by bulk migration, in fact equivalent to " static
Subtree ", because the subtree is will not be divided again.This mode avoids the excessively scrappy of metadata segmentation, and being conducive to will be right
Should be included in same subtree using the corresponding metadata of operation is accessed so that the metadata evolution that application is accessed is at one
Carry out in MDS, be not in the situation across MDS, effectively reduce the migration frequency of subtree.
Embodiment three
The present embodiment is also how description migrates out certain load value from certain MDS, but with the master of embodiment two
Distinguishing is, the present embodiment is also provided with the concept concurrently organized, on the basis of the efficiency for improving meta-data distribution management, to enter
One step improves the effect of load balancing.
The concept concurrently organized is illustrated first:In at least two subtrees that subtree flag property is set, select with identical
Multiple subtrees of load variation tendency are arranged as the root of each subtree in concurrently group, and the concurrently group
Concurrently organize attribute.On the one hand, concurrently group includes multiple subtrees, and the root of each subtree is to be provided with subtree mark
The catalogue of attribute.On the other hand, concurrently this multiple subtree in group the characteristics of be exactly to load variation tendency with identical.
Described identical load variation tendency is referred to for example, it is assumed that concurrently having two subtrees, the two subtrees in group
Load value be always substantially increased within certain time, or decline to a great extent within certain time, such as it is one little at certain
When interval in, the load value of the two subtrees all rises 10, or can also be that a subtree rises 9 load values, separately
One subtree rises 10 load values, i.e., only emphasize that the load value of two subtrees rises, and what it was risen is concrete
Numerical value can some deviations, and other subtree does not all change substantially;Or, the two subtrees is negative within certain time
Load value all have dropped 8 load values.
Identical load variation tendency generally indicates that the corresponding file of the two subtrees generally all can be accessed simultaneously, institute
With the change that just can occur loading simultaneously;By taking Fig. 4 as an example, for the subtree with catalogue k1_1 as root and with catalogue j1
For the subtree of root, the two subtrees just belong to a concurrently group, when user enable apply when, file k_ should be accessed
E.avi, can generally also access file james.avi, because both of these documents is associated, cause above-mentioned two
Always there is synchronous rising or decline in the load value of tree, and the meaning of this namely " concurrent ", user is applied at same section
Between access multiple subtrees in identical group simultaneously.
If having concurrently group in same MDS, show that the change of the load of the MDS can be very big, such as assume the MDS
It is middle to there is a concurrently group for including 5 subtrees, then it is possible that the load value of this 5 subtrees increases in certain time period
Plus so that significantly load value can quickly occur in the MDS to be increased, and the burden of MDS can be very big;So, the present embodiment arranges concurrent
Group attribute is distinguishing this kind of subtree, and the rule for arranging is, if to migrate subtree, prioritizing selection concurrently in group as far as possible
Tree is migrated, it is to avoid concurrently the subtree in group increases the burden of MDS in same MDS.
By taking the subtree that migrate 15 load values in Fig. 4 as an example, this is by search procedure and the embodiment two of the subtree of migration
It is identical, no longer in detail, only illustrate in search procedure with concurrently organize some relevant steps:Such as, it is provided with preferential lookup
During the catalogue of subtree mark, it is found that catalogue k1_1, catalogue j1 and catalogue j2 are all provided with subtree flag property, also,
The load value of catalogue k1_1 and catalogue j2 is all 15, now will judge to select the corresponding subtree of catalogue k1_1 or catalogue j2
Corresponding subtree.
Specifically, judge whether catalogue k1_1 and catalogue j2 have and concurrently organize attribute, if having concurrently organizing attribute, table
The bright subtree with the catalogue as root is a subtree concurrently in group, the prioritizing selection subtree.For example, catalogue k1_1 is tool
Having concurrently organize attribute, and the corresponding sub- tree and directory j1 of catalogue k1_1 corresponding subtrees are belonging to same concurrently organize;It is then excellent
The subtree with catalogue k1_1 as root is first selected to be migrated, the target directory also referred to as to be searched of catalogue k1_1.
Attribute is concurrently organized by arranging for catalogue, priority migration subtree concurrently in group can split the burden of MDS;Than
The corresponding subtrees of the corresponding sub- tree and directory j1 of catalogue k1_1 in Fig. 4 described above, if by the corresponding subtree of catalogue k1_1
Move out, even if the load variation tendency of the two subtrees is identical, such as load value increases in certain time, be also at two
Increase respectively in MDS, without the only increase in MDS1, so that the distribution being supported on before each MDS is more balanced.This reality
The set-up mode for concurrently organizing attribute for applying example is identical with the set-up mode of subtree flag property, repeats no more.
Further, multiple concurrently groups are there may exist in MDS, it is assumed that there are two concurrently all to there is suitable son in group
Tree, i.e. load attribute are identical with the load information that will be migrated, now the how Select Subtree in two concurrently group, and the present embodiment sets
Fixed rule is to determine the concurrently group that selected subtree is located, be in multiple concurrently groups including subtree quantity it is most and
Send out group;Such as, one of them concurrently group include 5 subtrees, another concurrently group include 2 subtrees, then prioritizing selection bag
Include the subtree in the concurrently group of 5 subtrees because including subtree it is more, when load rises, caused MDS burden is bigger.
The embodiment of the present invention for catalogue by arranging subtree flag property and concurrently organizes attribute so that the pipe of meta-data distribution
Reason is in hgher efficiency, and the effect of load balancing also more preferably, improves the overall performance of MDS clusters.
For example, by arranging subtree flag property, by with the corresponding subtree whole installation of the root of subtree flag property
In same MDS, it is possible to reduce across the operation of MDS, the change of metadata is performed inside MDS, which offers a saving across MDS
The time of information transfer during operation, MDS can just process more meta-data distribution management roles.
Referring to the MDS clusters overall performance contrast chart that Fig. 5 and Fig. 6, Fig. 5 are metadata management method embodiment of the present invention
One, Fig. 6 are that the MDS clusters overall performance of metadata management method embodiment of the present invention contrasts chart two, the vertical seat of the two tables
Mark all represents load value, from the figure, it can be seen that after using the scheme of the embodiment of the present invention (after improving), before enforcement
The load value of (before improving) indices is obtained for raising.Than document creation as illustrated in the drawing (file creation)
(file removal) load, directory creating (directory creation) load, directory delete are deleted in load, file
(directory removal) load, subtree create (tree creation) load and subtree is deleted (tree removal) and born
Carry, and file start (file start) load, catalogue start (directory start) load, the load described in these
It is that MDS clusters receive using the access operation corresponding load related to process, such as processes what file creation operation caused
Load.Increase than the above-mentioned load before improvement after improvement, the application access operation task that MDS clusters are processed after showing to improve is than it
Before increase, such as be only capable of processing 2 file creation operations before improving in certain time, 5 files wounds can be processed after improvement
Build and operate, hence it is evident that improve the performance of the MDS.
Again for example, attribute is concurrently organized by setting, the subtree that prioritizing selection belongs in same concurrently group is migrated, with
Multiple subtrees in concurrently group are distributed in multiple MDS, it is to avoid the MDS burdens for concentrating on same MDS and bringing
It is excessive.After such load balance process, the burden of MDS is alleviated, the metadata treatment effeciency of MDS will be improved, just
Higher application can be processed and access operation;For example, the MDS1 in Fig. 4 causes this due to wherein there is concurrently group before improvement
The heavier loads of MDS1, the treatment effeciency of MDS1 will be slack-off, and due to having carried out load balancing after improving, the burden of the MDS1
It is mitigated, improves metadata treatment effeciency, the task amount of process is also just more.
Referring to the MDS cluster load distributions before the improvement that Fig. 7 and Fig. 8, Fig. 7 are metadata management method embodiment of the present invention
Chart, Fig. 8 is the MDS cluster load distribution charts after the improvement of metadata management method embodiment of the present invention, the two tables
Ordinate all represents load value.From the figure, it can be seen that the not perfectly flat weighing apparatus of the load distribution of MDS0 and MDS1 before improving, block diagram
Shape is uneven, and the process load value that ordinate shows is relatively low, shows that process task amount is less;And improve after, MDS0 and
The load distribution of MDS1 has been equalized, and load balancing effect before improvement substantially than being improved, also, ordinate shows
Process to be there has also been before load value is relatively improved and greatly improve, improve the performance of MDS.
It should be noted that the embodiment of the present invention is with the MDS of cluster file system as object, meta-data distribution pipe is illustrated
The method of reason, but in being embodied as, it is not limited to MDS scenes, the method for the embodiment of the present invention can be used for other needs management
The system of file system metadata service distribution.
Example IV
Fig. 9 is the structural representation of the embodiment of meta data management device of the present invention, and the device can perform the present invention and appoint
The method of meaning embodiment, as shown in figure 9, the device can include:Load determining unit 91, sub-tree search unit 92 and subtree are moved
Move unit 93;Wherein,
Load determining unit 91, for obtaining the load information of migration;
Sub-tree search unit 92, for searching metadata in catalogue directory attribute, select directory attribute include with
The load information that the load information of migration is matched and the catalogue with subtree mark are target directory, it is determined that with described
Target directory is all subtrees of root as the target subtree that will be migrated;The subtree mark is set in advance in by the son
In the directory attribute of the root that tree mark is identified;
Subtree migration units 93, for by the target subtree bulk migration.
Further, the subtree of the root of the subtree mark mark includes that the minimum of Operational Visit opereating specification is single
The corresponding metadata of unit.
Further, it is included in visiting frequency in preset time period in the subtree of the root of the subtree mark mark to reach
To the metadata of predetermined threshold value.
Figure 10 is the structural representation of another embodiment of meta data management device of the present invention, and the present embodiment is being tied shown in Fig. 9
On the basis of structure, meta data management device also includes:Attribute setting unit 94, for searching first number in the sub-tree search unit
Before the directory attribute of the catalogue according in, in the directory attribute of root has at least two subtrees of subtree mark, select
Multiple subtrees with identical load variation tendency as a concurrent group, and by the root mesh of each subtree in the concurrently group
Concurrent group mark is set in the directory attribute of record;
The sub-tree search unit 92, specifically for the directory attribute of the catalogue in lookup metadata, selects directory attribute
In not only include the load information and with subtree mark, also including the concurrent group mark catalogue as the target mesh
All subtrees of the record determination using the target directory as root are used as the target subtree that will be migrated;The subtree mark sets in advance
Put in the directory attribute of the root identified by the subtree mark.
Further, sub-tree search unit 92, specifically for from the concurrently group most including subtree quantity, selecting catalogue
Not only include the load information and subtree mark in attribute, also including the concurrent group mark catalogue as the mesh
Heading is recorded.
Further, the meta data management device of the present embodiment, also includes:Attribute acquiring unit 95, for receive user
The attribute arranged by application programming interface API, the attribute includes the subtree mark and concurrent group mark.
Embodiment five
Figure 11 is the structural representation of the calculate node embodiment of metadata management of the present invention, and such as Figure 11, the present embodiment is carried
A kind of schematic diagram of calculate node 700 is supplied.The calculate node 700 possibly host server comprising computing capability, or
It is personal computer (personal computer, abbreviation:PC), or portable portable computer or terminal etc.,
The specific embodiment of the invention is not limited implementing for calculate node.
The calculate node 700 includes:Processor (processor) 710, communication interface (Communications
Interface) 720, memory (memory) 730, bus 740.Processor 710, communication interface 720, memory 730 are by total
Line 740 completes mutual communication.
Communication interface 720, for receiving program with net element communication.
Processor 710, for configuration processor 732.Specifically, program 732 can include program code, described program code
Including computer-managed instruction.
The computing device program 732 causes the calculate node of metadata management to perform the skill of any of the above-described embodiment of the method
Art scheme.
Processor 710 is probably a central processing unit (Central Processing Unit, abbreviation:CPU), or
It is specific integrated circuit (Application Specific Integrated Circuit, abbreviation:ASIC), or matched somebody with somebody
It is set to one or more integrated circuits for implementing the embodiment of the present invention.
Memory 730, for depositing program 732.Memory 730 may include high speed random access memory (random
Access memory, referred to as:RAM), it is also possible to also including nonvolatile memory (non-volatile memory), for example
At least one magnetic disc store.Program 732 specifically can include:
Load determining unit 91, for obtaining the load information of migration;
Sub-tree search unit 92, for searching metadata in catalogue directory attribute, select directory attribute include with
The load information that the load information of migration is matched and the catalogue with subtree mark are target directory, it is determined that with described
Target directory is all subtrees of root as the target subtree that will be migrated;The subtree mark is set in advance in by the son
In the directory attribute of the root that tree mark is identified;
Subtree migration units 93, for by the target subtree bulk migration.
Each unit is implemented referring to the corresponding units in Fig. 9-embodiment illustrated in fig. 10 in program 732, and here is not gone to live in the household of one's in-laws on getting married
State.
Further, the subtree of the root of the subtree mark mark includes that the minimum of Operational Visit opereating specification is single
The corresponding metadata of unit.
Further, it is included in visiting frequency in preset time period in the subtree of the root of the subtree mark mark to reach
To the metadata of predetermined threshold value.
Further, described program also includes:
Attribute setting unit, before the directory attribute for the catalogue in sub-tree search unit lookup metadata,
In the directory attribute of root has at least two subtrees of subtree mark, select multiple with identical load variation tendency
Subtree will arrange concurrently group mark as a concurrent group in the directory attribute of the root of each subtree in the concurrently group
Will;
The sub-tree search unit, specifically for the directory attribute of the catalogue in lookup metadata, in selecting directory attribute
Not only include the load information and with subtree mark, also including the concurrent group mark catalogue as the target directory
It is determined that all subtrees using the target directory as root are used as the target subtree that will be migrated;The subtree mark pre-sets
In the directory attribute of the root identified by the subtree mark.
Further, the sub-tree search unit, specifically for from the concurrently group most including subtree quantity, selecting mesh
Not only include the load information and subtree mark in record attribute, also including the concurrent group mark catalogue as described
Target directory.
Further, described program also includes:
Attribute acquiring unit, for the attribute that receive user is arranged by application programming interface API, the attribute bag
Include the subtree mark and concurrent group mark.
Those skilled in the art can be understood that, for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, may be referred to the corresponding process in preceding method embodiment, will not be described here.
In several embodiments provided herein, it should be understood that disclosed system, apparatus and method, can be with
Realize by another way.For example, device embodiment described above is only schematic, for example, the unit
Divide, only a kind of division of logic function can have other dividing mode, such as multiple units or component when actually realizing
Can with reference to or be desirably integrated into another system, or some features can be ignored, or not perform.It is another, it is shown or
The coupling each other for discussing or direct-coupling or communication connection can be by some communication interfaces, between device or unit
Connect coupling or communicate to connect, can be electrical, mechanical or other forms.
The unit as separating component explanation can be or may not be it is physically separate, it is aobvious as unit
The part for showing can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple
On NE.Some or all of unit therein can according to the actual needs be selected to realize the mesh of this embodiment scheme
's.
In addition, each functional unit in each embodiment of the invention can be integrated in a processing unit, it is also possible to
It is that unit is individually physically present, it is also possible to which two or more units are integrated in a unit.
If the function is realized and as independent production marketing or when using using in the form of SFU software functional unit, can be with
In being stored in a computer read/write memory medium.Based on such understanding, technical scheme is substantially in other words
The part contributed to prior art or the part of the technical scheme can be embodied in the form of software product, the meter
Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be individual
People's computer, server, or network equipment etc.) perform all or part of step of each embodiment methods described of the invention.
And aforesaid storage medium includes:USB flash disk, portable hard drive, read-only storage (Read-Only Memory, abbreviation:ROM it is), random
Access memory (Random Access Memory, referred to as:RAM), magnetic disc or CD etc. are various can be with store program codes
Medium.
Finally it should be noted that:Various embodiments above only to illustrate technical scheme, rather than a limitation;To the greatest extent
Pipe has been described in detail with reference to foregoing embodiments to the present invention, it will be understood by those within the art that:Its according to
So the technical scheme described in foregoing embodiments can be modified, either which part or all technical characteristic are entered
Row equivalent;And these modifications or replacement, do not make the essence disengaging various embodiments of the present invention technology of appropriate technical solution
The scope of scheme.
Claims (13)
1. a kind of metadata management method, it is characterised in that include:
Obtain the load information of migration;
The directory attribute of the catalogue in metadata is searched, selects directory attribute to include and the load information phase by migration
The load information matched somebody with somebody and the catalogue with subtree mark are target directory, it is determined that all sons with the target directory as root
Set as the target subtree that will be migrated;The subtree mark is set in advance in the mesh of the root identified by the subtree mark
In record attribute;
By the target subtree bulk migration;
Wherein, the subtree mark is intended to indicate that all subtrees with the target directory as root are only capable of in an integral manner
Migration.
2. metadata management method according to claim 1, it is characterised in that the root of the subtree mark mark
Subtree includes the corresponding metadata of the minimum unit of Operational Visit opereating specification.
3. metadata management method according to claim 1, it is characterised in that the root of the subtree mark mark
The metadata that visiting frequency in preset time period reaches predetermined threshold value is included in subtree.
4. metadata management method according to claim 1, it is characterised in that the catalogue of the catalogue in metadata is searched
Before attribute, also include:
In the directory attribute of root has at least two subtrees of subtree mark, select with identical load variation tendency
Multiple subtrees will arrange concurrent as a concurrent group in the directory attribute of the root of each subtree in the concurrently group
Group mark;
The selection directory attribute is included with the load information that the load information of migration matches and with subtree mark
The catalogue of will is target directory, including:Select not only to include the load information in directory attribute and with subtree mark, also wrap
The catalogue of the concurrent group mark is included as the target directory.
5. metadata management method according to claim 4, it is characterised in that not only include in the selection directory attribute
The load information and with subtree mark, also including the concurrent group mark catalogue as the target directory, including:
From the concurrently group most including subtree quantity, select not only to include the load information and subtree in directory attribute
Mark, also including the concurrent group mark catalogue as the target directory.
6. metadata management method according to claim 4, it is characterised in that the directory attribute is that receive user passes through
The attribute that application programming interface API is arranged, the attribute includes the subtree mark and concurrent group mark.
7. a kind of meta data management device, it is characterised in that include:
Load determining unit, for obtaining the load information of migration;
Sub-tree search unit, for searching metadata in catalogue directory attribute, select directory attribute include and it is described will
The load information that the load information of migration matches and the catalogue with subtree mark are target directory, it is determined that with the target mesh
Record all subtrees for root as will migration target subtree;The subtree mark is set in advance in by the subtree mark
In the directory attribute of the root for identifying;
Subtree migration units, for by the target subtree bulk migration;
Wherein, the subtree mark is intended to indicate that all subtrees with the target directory as root are only capable of in an integral manner
Migration.
8. meta data management device according to claim 7, it is characterised in that the root of the subtree mark mark
Subtree includes the corresponding metadata of the minimum unit of Operational Visit opereating specification.
9. meta data management device according to claim 7, it is characterised in that the root of the subtree mark mark
The metadata that visiting frequency in preset time period reaches predetermined threshold value is included in subtree.
10. meta data management device according to claim 7, it is characterised in that also include:
Attribute setting unit, for before the directory attribute that the catalogue in metadata is searched in the sub-tree search unit, in root
The directory attribute of catalogue has at least two subtrees of subtree mark, selects the multiple subtrees with identical load variation tendency
As a concurrent group, and concurrent group mark will be set in the directory attribute of the root of each subtree in the concurrently group;
The sub-tree search unit, specifically for the directory attribute of the catalogue in lookup metadata, selects in directory attribute not only
Including the load information and with subtree mark, also determine as the target directory including the catalogue of the concurrent group mark
All subtrees using the target directory as root are used as the target subtree that will be migrated;The subtree mark be set in advance in by
In the directory attribute of the root that the subtree mark is identified.
11. meta data management devices according to claim 10, it is characterised in that
The sub-tree search unit, specifically for from the concurrently group most including subtree quantity, selecting in directory attribute not only
Including the load information and with subtree mark, also including the concurrent group mark catalogue as the target directory.
12. meta data management devices according to claim 10, it is characterised in that also include:
Attribute acquiring unit, for the attribute that receive user is arranged by application programming interface API, the attribute includes institute
State subtree mark and concurrent group mark.
13. a kind of calculate nodes of metadata management, it is characterised in that the calculate node includes:Processor, communication interface,
Memory and bus;Wherein, the processor, communication interface and memory complete mutual communication by the bus;
The communication interface, the calculate node for the metadata management receives program;
The processor, for configuration processor;
The memory, for depositing program;
The computing device described program causes the calculate node perform claim of the metadata management to require appointing in 1 to 6
One method.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2012/085344 WO2014082203A1 (en) | 2012-11-27 | 2012-11-27 | Metadata management method and device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN103688257A CN103688257A (en) | 2014-03-26 |
CN103688257B true CN103688257B (en) | 2017-04-26 |
Family
ID=50323329
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201280002998.8A Active CN103688257B (en) | 2012-11-27 | 2012-11-27 | Method and device for managing metadata |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN103688257B (en) |
WO (1) | WO2014082203A1 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106326040B (en) * | 2016-08-27 | 2019-12-31 | 苏州浪潮智能科技有限公司 | Snapshot metadata management method and device |
CN106446197B (en) * | 2016-09-30 | 2019-11-19 | 华为数字技术(成都)有限公司 | A kind of date storage method, apparatus and system |
CN106777062B (en) * | 2016-12-12 | 2020-03-10 | 郑州云海信息技术有限公司 | Method and device for managing metadata |
CN107480310B (en) * | 2017-09-29 | 2020-09-04 | 郑州云海信息技术有限公司 | Dynamic load balancing method and system for metadata cluster directory |
CN107798104A (en) * | 2017-10-31 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of catalog management method, device, equipment and computer-readable recording medium |
CN112181899A (en) * | 2019-07-05 | 2021-01-05 | 中兴通讯股份有限公司 | Metadata processing method and device and computer readable storage medium |
CN110932935A (en) * | 2019-11-26 | 2020-03-27 | 深圳前海微众银行股份有限公司 | Resource control method, device, equipment and computer storage medium |
CN113055448B (en) * | 2021-02-28 | 2023-03-28 | 新华三信息技术有限公司 | Metadata management method and device |
CN112948354A (en) * | 2021-03-01 | 2021-06-11 | 北京金山云网络技术有限公司 | Method and device for creating copy cluster, electronic device and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
CN102055650A (en) * | 2009-10-29 | 2011-05-11 | 华为技术有限公司 | Load balance method and system and management server |
CN102523158A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Metadata server cluster load balancing method based on weight |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4230673B2 (en) * | 2001-02-22 | 2009-02-25 | 富士通株式会社 | Service management device |
CN101697168B (en) * | 2009-10-22 | 2011-10-19 | 中国科学技术大学 | Method and system for dynamically managing metadata of distributed file system |
CN102571772B (en) * | 2011-12-26 | 2014-08-27 | 华中科技大学 | Hot spot balancing method for metadata server |
-
2012
- 2012-11-27 WO PCT/CN2012/085344 patent/WO2014082203A1/en active Application Filing
- 2012-11-27 CN CN201280002998.8A patent/CN103688257B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
CN102055650A (en) * | 2009-10-29 | 2011-05-11 | 华为技术有限公司 | Load balance method and system and management server |
CN102523158A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Metadata server cluster load balancing method based on weight |
Also Published As
Publication number | Publication date |
---|---|
CN103688257A (en) | 2014-03-26 |
WO2014082203A1 (en) | 2014-06-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN103688257B (en) | Method and device for managing metadata | |
CN100468402C (en) | Sort data storage and split catalog inquiry method based on catalog tree | |
CN104583971B (en) | Management system and management method | |
CN104067216B (en) | System and method for implementing expansible data storage service | |
AU2002312508B2 (en) | Storage system having partitioned migratable metadata | |
CN101110846B (en) | Management method and equipment for MAC address table item | |
CN103544156B (en) | File memory method and device | |
CN108509275B (en) | A kind of catalogue moving method and metadata load-balancing method | |
CN107111450A (en) | The disk partition splicing and equilibrium again carried out using partition table | |
US20160283538A1 (en) | Fast multi-tier indexing supporting dynamic update | |
WO2017065885A1 (en) | Distributed pipeline optimization data preparation | |
CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
CN104111924B (en) | A kind of Database Systems | |
CN104081391A (en) | Single instantiation method using file clone and file storage system utilizing the same | |
CN103229173A (en) | Metadata management method and system | |
CN100424699C (en) | Attribute extensible object file system | |
CN105608228B (en) | A kind of efficient distributed RDF data storage method | |
JP2018538600A (en) | Signature-based cache optimization for data preparation | |
CN103246549A (en) | Method and system for data transfer | |
Liu et al. | An improved hadoop data load balancing algorithm | |
WO2017065888A1 (en) | Step editor for data preparation | |
CN109508319A (en) | A kind of cross-platform resource management system of 3D and method | |
CN109885642A (en) | Classification storage method and device towards full-text search | |
CN103123651B (en) | A kind of method, device and mobile device checking multiple same class file | |
CN107423321B (en) | Method and device suitable for cloud storage of large-batch small files |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |