CN106777062A - A kind of method and device for managing metadata - Google Patents

A kind of method and device for managing metadata Download PDF

Info

Publication number
CN106777062A
CN106777062A CN201611139129.6A CN201611139129A CN106777062A CN 106777062 A CN106777062 A CN 106777062A CN 201611139129 A CN201611139129 A CN 201611139129A CN 106777062 A CN106777062 A CN 106777062A
Authority
CN
China
Prior art keywords
packet
virtual directory
metadata
virtual
currently
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201611139129.6A
Other languages
Chinese (zh)
Other versions
CN106777062B (en
Inventor
李雪生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhengzhou Yunhai Information Technology Co Ltd
Original Assignee
Zhengzhou Yunhai Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhengzhou Yunhai Information Technology Co Ltd filed Critical Zhengzhou Yunhai Information Technology Co Ltd
Priority to CN201611139129.6A priority Critical patent/CN106777062B/en
Publication of CN106777062A publication Critical patent/CN106777062A/en
Application granted granted Critical
Publication of CN106777062B publication Critical patent/CN106777062B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/188Virtual file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots
    • G06F16/119Details of migration of file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/164File meta data generation

Abstract

The invention provides a kind of method and device for managing metadata, the method includes:For corresponding at least two packet of catalogue, the corresponding mapping range of each packet is determined;The corresponding metadata of each file is stored into packet corresponding with the mapping range that it matches respectively;When the memory capacity for monitoring catalogue is not located in preset threshold range, the virtual directory of the first quantity is set up, and according to packet migration rules, corresponding migration process is carried out for the whole packets that there is currently.Based on packet to store magnanimity metadata respectively, therefore when the memory capacity of catalogue transfinites, can be by way of increasing virtual directory and migrating packet, with by the grouped storage metadata of virtual directory, such that it is able to solve the problems, such as that catalogue memory capacity transfinites.Therefore, this programme can improve the recall precision of metadata.

Description

A kind of method and device for managing metadata
Technical field
The present invention relates to field of computer technology, more particularly to a kind of method and device for managing metadata.
Background technology
As the development in big data epoch, non-structured data are in explosive growth, data yield increasingly explodes.Example Such as, large-scale sensor can produce the data slot of magnanimity, so that the huge small data file of quantity of formation.Wherein it is possible to adopt The mode of data is preserved to store the mass data of generation with file, in order to data sharing and management.
At present, for the corresponding metadata of file, the metadata of magnanimity can be stored in the single mesh of file system In record.
But, with being constantly stored in for each metadata, the quantity of metadata is easily excessive in single catalogue.Therefore work as and require to look up When file destination is retrieved with carrying out respective meta-data, existing metadata management mode can reduce the recall precision of metadata.
The content of the invention
The invention provides a kind of method and device for managing metadata, it is possible to increase the recall precision of metadata.
In order to achieve the above object, the present invention is achieved through the following technical solutions:
On the one hand, the invention provides a kind of method for managing metadata, including:
S1:For corresponding at least two packet of catalogue, each corresponding mapping range of the packet is determined;
S2:It is performed both by for the corresponding metadata of each file:The metadata is stored to the mapping matched with it In the corresponding packet of scope;
S3:When the memory capacity for monitoring the catalogue is not located in the range of first threshold set in advance, S4 is performed;
S4:The virtual directory of the first quantity is set up, and according to predetermined packet migration rules, for what be there is currently All packet, carries out corresponding migration process.
Further, the title of this document is included in the corresponding metadata of each file;
The S2, including:It is performed both by for the corresponding metadata of each file:According to including file name Claim, calculate the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, wherein, the Kazakhstan Uncommon value is located in the target mapping range;The metadata is stored into the corresponding packet of the target mapping range.
Further, the memory capacity coincidence formula one of the catalogue, wherein,
The formula one includes:
Wherein, X is the memory capacity of the catalogue, and n is the number of at least two packet, NiIt is described at least two The number of the metadata stored in i-th packet in packet.
Further, first quantity is by the migration process, so that any virtual directory that there is currently Memory capacity is respectively positioned on the minimum number in the range of the first threshold.
Further, also include after S4:Monitoring that it is described that the memory capacity that there is any virtual directory is not located at When in the range of first threshold, S4 is performed.
Further, the packet migration rules, including:Each order of packets arrangement in the whole packets that there is currently, And in the whole virtual directories that there is currently each virtual directory order arrange when, according to formula two determine each virtual directory The number of corresponding packet;
When the number of the whole virtual directories that there is currently is first quantity, according to each the virtual mesh determined The number of corresponding packet is recorded, described at least two packets are migrated into each virtual directory that there is currently respectively, its In, for any two virtual directory, when second virtual directory that is located therein of putting in order of the first virtual directory therein Put in order after when, the putting in order of each packet in first virtual directory is respectively positioned on each in second virtual directory Packet put in order after;
When the number of the whole virtual directories that there is currently not is first quantity, the arrangement based on each virtual directory is suitable Sequence, is performed both by for each virtual directory successively:The number for calculating the current corresponding packet of the virtual directory is subtracted and determined The corresponding packet of the virtual directory number difference;When it is positive number to judge the difference, the virtual directory pair is determined The packet to be migrated answered, wherein, in the current corresponding all packets of the virtual directory, the position that puts in order of the packet to be migrated After other packets, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order It is arranged in the next virtual directory after the virtual directory;
The formula two includes:
Wherein, X is the number of the whole packets that there is currently;N is the number of the whole virtual directories that there is currently;x1For In the whole virtual directories that there is currently, the number of the corresponding packet of any virtual directory in addition to last position virtual directory, x2For The number of the corresponding packet of the last position virtual directory.
Further, also include after S4:Monitoring the last position virtual directory and virtual directory adjacent thereto Packet it is unbalanced degree be located at Second Threshold set in advance in the range of when, set up correspond to the last position virtual directory extremely A few packet, so that the unbalanced degree of the packet is located in the range of the Second Threshold;For the whole packet that there is currently, Redefine each and be grouped corresponding mapping range;It is performed both by for the corresponding metadata of each file:Judge Whether packet that the metadata is currently located and packet corresponding with the mapping range that it matches are identical, if it is not, by this yuan of number According to migrating into packet corresponding with the mapping range that it matches.
Further, the number of at least one packet of the foundation is to meet described so that the unbalanced degree of packet Minimum number in the range of the Second Threshold.
Further, the unbalanced degree coincidence formula three of the packet, wherein,
The formula three includes:
Wherein, Y is the unbalanced degree of packet, XnIt is the number of the corresponding packet of the last position virtual directory, XmIt is and institute State the number of the adjacent corresponding packet of virtual directory of last position virtual directory.
On the other hand, the invention provides a kind of device for managing metadata, including:
Determining unit, for for corresponding at least two packet of catalogue, determining each corresponding mapping of the packet Scope;
Map unit, for being performed both by for the corresponding metadata of each file:By the metadata store to its phase In the corresponding packet of mapping range of matching, and trigger the first monitoring unit;
First monitoring unit, for not being located at the first threshold set in advance in the memory capacity for monitoring the catalogue When in the range of value, processing unit is triggered;
The processing unit, the virtual directory for setting up the first quantity, and according to predetermined packet migration rules, For the whole packet that there is currently, corresponding migration process is carried out.
Further, the title of this document is included in the corresponding metadata of each file;
The map unit, specifically for being performed both by for the corresponding metadata of each file:According to including The title of file, calculates the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, its In, the cryptographic Hash is located in the target mapping range;The metadata is stored to corresponding point of the target mapping range In group.
Further, the memory capacity coincidence formula one of the catalogue, wherein,
The formula one includes:
Wherein, X is the memory capacity of the catalogue, and n is the number of at least two packet, NiIt is described at least two The number of the metadata stored in i-th packet in packet.
Further, first quantity is by the migration process, so that any virtual directory that there is currently Memory capacity is respectively positioned on the minimum number in the range of the first threshold.
Further, first monitoring unit, is additionally operable in the memory capacity for monitoring to have any virtual directory not When in the range of the first threshold, the processing unit is triggered.
Further, the packet migration rules, including:Each order of packets arrangement in the whole packets that there is currently, And in the whole virtual directories that there is currently each virtual directory order arrange when, according to formula two determine each virtual directory The number of corresponding packet;
When the number of the whole virtual directories that there is currently is first quantity, according to each the virtual mesh determined The number of corresponding packet is recorded, described at least two packets are migrated into each virtual directory that there is currently respectively, its In, for any two virtual directory, when second virtual directory that is located therein of putting in order of the first virtual directory therein Put in order after when, the putting in order of each packet in first virtual directory is respectively positioned on each in second virtual directory Packet put in order after;
When the number of the whole virtual directories that there is currently not is first quantity, the arrangement based on each virtual directory is suitable Sequence, is performed both by for each virtual directory successively:The number for calculating the current corresponding packet of the virtual directory is subtracted and determined The corresponding packet of the virtual directory number difference;When it is positive number to judge the difference, the virtual directory pair is determined The packet to be migrated answered, wherein, in the current corresponding all packets of the virtual directory, the position that puts in order of the packet to be migrated After other packets, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order It is arranged in the next virtual directory after the virtual directory;
The formula two includes:
Wherein, X is the number of the whole packets that there is currently;N is the number of the whole virtual directories that there is currently;x1For In the whole virtual directories that there is currently, the number of the corresponding packet of any virtual directory in addition to last position virtual directory, x2For The number of the corresponding packet of the last position virtual directory.
Further, the device of the management metadata also includes:Unit is set up in second monitoring unit, packet;
Second monitoring unit, for monitor the last position virtual directory and virtual directory adjacent thereto point The unbalanced degree of group is not located at when in the range of Second Threshold set in advance, is triggered the packet and is set up unit;
Unit is set up in the packet, for setting up at least one packet corresponding to the last position virtual directory, so that institute State the unbalanced degree of packet to be located in the range of the Second Threshold, and trigger the determining unit;
The determining unit, be additionally operable to receive it is described packet set up the trigger signal that unit is sent when, for current Whole packets of presence, redefine each and are grouped corresponding mapping range, and trigger the map unit;
The map unit, is additionally operable to when the trigger signal that the determining unit is sent is received, for each text The corresponding metadata of part is performed both by:Judge packet that the metadata is currently located and corresponding with the mapping range that it matches point Whether group is identical, if it is not, by the metadata migration to packet corresponding with the mapping range that it matches.
Further, the number that at least one packet of unit foundation is set up in the packet is to meet described so that described Minimum number of the unbalanced degree of packet in the range of the Second Threshold.
Further, the unbalanced degree coincidence formula three of the packet, wherein,
The formula three includes:
Wherein, Y is the unbalanced degree of packet, XnIt is the number of the corresponding packet of the last position virtual directory, XmIt is and institute State the number of the adjacent corresponding packet of virtual directory of last position virtual directory.
The invention provides a kind of method and device for managing metadata, it is grouped for catalogue corresponding at least two, really Determine the corresponding mapping range of each packet;The corresponding metadata of each file is stored to the mapping range matched with it respectively In corresponding packet;When the memory capacity for monitoring catalogue is not located in preset threshold range, the virtual of the first quantity is set up Catalogue, and according to packet migration rules, corresponding migration process is carried out for the whole packets that there is currently.Based on packet distinguishing Storage magnanimity metadata, therefore when the memory capacity of catalogue transfinites, can by way of increasing virtual directory and migrating packet, With by the grouped storage metadata of virtual directory, such that it is able to solve the problems, such as that catalogue memory capacity transfinites.Therefore, the present invention The recall precision of metadata can be improved.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are the present invention Some embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can also basis These accompanying drawings obtain other accompanying drawings.
Fig. 1 is a kind of flow chart of the method for management metadata that one embodiment of the invention is provided;
Fig. 2 is a kind of schematic diagram of the framework of management metadata that one embodiment of the invention is provided;
Fig. 3 is the flow chart of the method for another management metadata that one embodiment of the invention is provided;
Fig. 4 is a kind of schematic diagram of the device of management metadata that one embodiment of the invention is provided;
Fig. 5 is the schematic diagram of the device of another management metadata that one embodiment of the invention is provided.
Specific embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is A part of embodiment of the present invention, rather than whole embodiments, based on the embodiment in the present invention, those of ordinary skill in the art The every other embodiment obtained on the premise of creative work is not made, belongs to the scope of protection of the invention.
As shown in figure 1, the embodiment of the invention provides a kind of method for managing metadata, may comprise steps of:
Step 101:For corresponding at least two packet of catalogue, each corresponding mapping range of the packet is determined.
Step 102:It is performed both by for the corresponding metadata of each file:The metadata is stored to it and is matched In the corresponding packet of mapping range.
Step 103:When the memory capacity for monitoring the catalogue is not located in the range of first threshold set in advance, hold Row step 104.
Step 104:The virtual directory of the first quantity is set up, and according to predetermined packet migration rules, for current Whole packets of presence, carry out corresponding migration process.
A kind of method for managing metadata is the embodiment of the invention provides, is grouped for catalogue corresponding at least two, really Determine the corresponding mapping range of each packet;The corresponding metadata of each file is stored to the mapping range matched with it respectively In corresponding packet;When the memory capacity for monitoring catalogue is not located in preset threshold range, the virtual of the first quantity is set up Catalogue, and according to packet migration rules, corresponding migration process is carried out for the whole packets that there is currently.Based on packet distinguishing Storage magnanimity metadata, therefore when the memory capacity of catalogue transfinites, can by way of increasing virtual directory and migrating packet, With by the grouped storage metadata of virtual directory, such that it is able to solve the problems, such as that catalogue memory capacity transfinites.Therefore, the present invention Embodiment can improve the recall precision of metadata.
In detail, for each file in file system, two parts, the respectively corresponding unit of file be may each comprise The content of data and file.Wherein, in the embodiment of the present invention, the metadata of each file can be carried out by catalogue to unify pipe Reason.
In detail, for the catalogue of file system, the catalogue to that should have multiple packets, and can respectively be grouped corresponding mapping Scope can be stored in each metadata correspondence mappings to respective packets and is managed with difference such that it is able to make.
In detail, the corresponding metadata of file can include title, time, authority, extended attribute, the storage location of file Deng.
Therefore, in an embodiment of the invention, in order to illustrate a kind of possibility reality that metadata is mapped to respective packets Existing mode, so, the title of this document is included in the corresponding metadata of each file;
The step 102, including:It is performed both by for the corresponding metadata of each file:According to including file Title, calculate the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, wherein, institute Cryptographic Hash is stated to be located in the target mapping range;The metadata is stored into the corresponding packet of the target mapping range.
As an example it is assumed that the catalogue of file system is current to that should have 10 packets, and this 10 are grouped corresponding mapping Scope be followed successively by [0,10], (10,20] ..., (90,100].Metadata corresponding for any file, according to including This document title, its corresponding cryptographic Hash can be calculated according to the title.Assuming that for a certain metadata, the Kazakhstan for calculating Uncommon value is 17, due to the 17 corresponding mapping ranges of second packet that can be fallen into above-mentioned 10 packets:(10,20], explanation The metadata and mapping range:(10,20] match, therefore deposited in the metadata being mapped into this second packet Storage.In this way, based on same realization principle, each metadata can be respectively mapped to the mapping range pair that matches with it In the packet answered.
With the continuous generation of metadata, each metadata is constantly stored in respective packets, therefore in the corresponding each packet of catalogue The quantity of the metadata for being stored is continuously increased.When incrementss reach to a certain degree, the memory capacity of catalogue can be caused to surpass Limit.When the memory capacity of catalogue transfinites, the execution of the operations such as retrieval, the cache management of metadata is not conducive to, therefore can be by increasing Plus the implementation of virtual directory shares the storage pressure of catalogue.
In detail, by increasing virtual directory, the corresponding all packets of catalogue can be migrated to newly-increased virtual directory In, to alleviate the storage pressure of catalogue.Wherein, when the memory capacity of catalogue transfinites, if only increase a virtual directory, and will The corresponding all packets of catalogue are migrated into the virtual directory, and the memory capacity for also resulting in the virtual directory transfinites.Cause This, when the memory capacity of catalogue transfinites, that is, when needing to set up virtual directory first, acquiescence needs to increase by least two virtual mesh Record.Therefore, the first quantity in above-mentioned steps 104 can be at least two.
For example, catalogue is current to that should have 10 packets, if the memory capacity of catalogue transfinites, can increase two virtual directories newly, And migrate into two virtual directories this 10 packets respectively, so that each virtual directory both corresponds to 5 packets.Due to inciting somebody to action Corresponding 10 packets of catalogue are moved out, and the storage pressure of catalogue can be made to alleviate significantly, so as to solve what its memory capacity transfinited Problem.Meanwhile, the memory capacity of each virtual directory does not transfinite currently.
Accordingly, when setting up of any virtual directory is completed, the corresponding metadata of the virtual directory can be being generated.The present invention In one embodiment, it is preferable that the corresponding metadata of the virtual directory of generation can be stored into catalogue, with by catalogue pair Each virtual directory is managed collectively.Wherein, each virtual directory of foundation can be as the subdirectory of catalogue, with by mesh Record is managed.
Based on the above, in an embodiment of the invention, it is preferable that first quantity is by the migration Reason, so that the memory capacity of any virtual directory that there is currently is respectively positioned on the minimum number in the range of the first threshold.
For example, being constantly stored in metadata, it is assumed that the current memory capacity of catalogue has reached 120,000, and sets Fixed first threshold scope is:≤ 10 ten thousand, illustrate that the current memory capacity of catalogue transfinites, therefore at least need to increase by two virtually Catalogue.For example, increase by two virtual directories and perform packet migration with divide equally packet after, the memory capacity of each newly-increased virtual directory 60,000 are can be about, both memory capacity does not transfinite.
Under normal circumstances, the cloud data writing rate in practical application, when the memory capacity of catalogue transfinites, that is, works as head Secondary when setting up virtual directory, the minimum number of virtual directory to be increased is usually 2.
Certainly, for some limit or special circumstances, it is assumed that the metadata amount of being stored at a certain moment is excessive, such as work as catalogue Preceding memory capacity has reached 220,000, each new if equally increasing by two virtual directories and performing packet migration to divide packet equally The memory capacity for increasing virtual directory can be about 110,000, and both memory capacity transfinites, or at least have one virtually Catalogue transfinites.Therefore, the minimum number of virtual directory to be increased should be 3.
Therefore, based on the above as can be seen that when virtual directory is set up first, the minimum number of above-mentioned first quantity It is 2.
Generally speaking, when the memory capacity that there is any virtual directory transfinites, the number of newly-established virtual directory should After ensure that execution packet migration operation, the memory capacity of each virtual directory does not transfinite in file system.
In an embodiment of the invention, the memory capacity meets following formula (1);
Wherein, X is the memory capacity of the catalogue, and n is the number of at least two packet, NiIt is described at least two The number of the metadata stored in i-th packet in packet.
As can be seen that the memory capacity of catalogue can be, the metadata of storage is individual in corresponding each packet of catalogue It is several plus and.
Accordingly, based on same realization principle, same to above-mentioned formula (1), in an embodiment of the invention, for any Virtual directory, the memory capacity of the virtual directory can be, the metadata stored in corresponding each packet of the virtual directory Number plus and.
After newly-built virtual directory, magnanimity metadata can be stored by the way that virtual directory is grouped.Certainly, with The continuous of newly-increased metadata is stored in, and the memory capacity of each virtual directory equally occurs overrun condition.
As shown in the above, when the memory capacity of catalogue transfinites, can be by increasing virtual directory and being grouped Migrate to solve the memory capacity overrun issues of catalogue.Similarly, when the memory capacity that there is any virtual directory transfinites, equally Can be by increasing virtual directory and carrying out packet migration to solve memory capacity overrun issues.Wherein, for depositing each time Storage capacity overrun issues, can be with newly-built at least one virtual directory.In this way, when it is non-set up virtual directory first when, above-mentioned The minimum number of one quantity is 1.
Therefore, in an embodiment of the invention, further include after step 104:Monitoring there is any void The memory capacity for intending catalogue is not located at when in the range of the first threshold, performs step 104.
In detail, the memory capacity simply by the presence of any virtual directory transfinites, you can perform step 104, with newly-increased virtual Catalogue simultaneously carries out packet migration, so that the memory capacity of each virtual directory does not transfinite.
In detail, for any virtual directory, by performing step 104, part therein packet can be moved out, to subtract The number of few its corresponding packet, therefore its memory capacity can be reduced, so as to solve the problems, such as that virtual directory memory capacity transfinites.
In an embodiment of the invention, in order to illustrate a kind of possible packet migration implementation, so, the packet Migration rules, including:Each order of packets arrangement in the whole packets that there is currently, and in the whole virtual directories that there is currently Each virtual directory order arrange when, the number of the corresponding packet of each virtual directory is determined according to following formula (2);
When the number of the whole virtual directories that there is currently is first quantity, according to each the virtual mesh determined The number of corresponding packet is recorded, described at least two packets are migrated into each virtual directory that there is currently respectively, its In, for any two virtual directory, when second virtual directory that is located therein of putting in order of the first virtual directory therein Put in order after when, the putting in order of each packet in first virtual directory is respectively positioned on each in second virtual directory Packet put in order after;
When the number of the whole virtual directories that there is currently not is first quantity, the arrangement based on each virtual directory is suitable Sequence, is performed both by for each virtual directory successively:The number for calculating the current corresponding packet of the virtual directory is subtracted and determined The corresponding packet of the virtual directory number difference;When it is positive number to judge the difference, the virtual directory pair is determined The packet to be migrated answered, wherein, in the current corresponding all packets of the virtual directory, the position that puts in order of the packet to be migrated After other packets, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order It is arranged in the next virtual directory after the virtual directory;
Wherein, X is the number of the whole packets that there is currently;N is the number of the whole virtual directories that there is currently;x1For In the whole virtual directories that there is currently, the number of the corresponding packet of any virtual directory in addition to last position virtual directory, x2For The number of the corresponding packet of the last position virtual directory.
In an embodiment of the invention, Fig. 2 can be expressed as a kind of framework for managing metadata.The framework can be represented It is to be currently included 20 packets, virtual directory 1 and virtual directories 2 under the catalogue of file system, and catalogue.Wherein, virtual mesh Record 1 corresponds to packet 1~packet 10, and virtual directory 2 corresponds to packet 11~packet 20.Additionally, for each packet, wherein Be stored with the metadata (not shown in fig. 2) matched with it.
In detail, for above-mentioned packet migration rules, Fig. 2 is refer to, for example:Assuming that being deposited in the catalogue of file system In 20 packets, 1~packet 20 is respectively grouped, and this 20 packets initially both correspond to catalogue.When the memory capacity of catalogue When transfiniting, it is generally the case that 2 virtual directories can be set up:Virtual directory 1 and virtual directory 2.In detail, for foundation Each virtual directory, is managed collectively by catalogue.
In fig. 2, above-mentioned 20 order of packets arrangement, virtual directory 1 is the first virtual directory, and virtual directory 2 is order The second virtual directory of arrangement.According to above-mentioned formula (2), grouping strategy can be:19 packets of the correspondence of virtual directory 1 and void Intend catalogue 2 correspondence 1 packet, virtual directory 1 correspondence 18 packet and virtual directory 2 correspondence 2 packet ..., virtual directory 1 10 packets of 10 packets of correspondence and the correspondence of virtual directory 2.
Wherein, because the metadata quantity stored in each packet is generally more or less the same, therefore no matter any of the above-described packet is used Strategy, after can ensureing packet migration, the memory capacity of virtual directory 1 and virtual directory 2 does not transfinite.But, when both are right When the packet number answered has big difference, both unbalanced degree of packet are higher, can influence the metadata management of whole file system Stability.
It is therefore preferred that can as much as possible choose both minimum grouping strategies of corresponding packet number difference, i.e., it is empty Intend catalogue 1 and virtual directory 2 and correspond to 10 packets.Certainly, based on different practical application requests, other can equally be used Grouping strategy, so that after packet migration, the memory capacity of each virtual directory does not transfinite.
10 grouping strategies of packet are corresponded to for virtual directory 1 and virtual directory 2,1~packet 10 can will be grouped, Migrated from catalogue into virtual directory 1, and 11~packet 20 will be grouped, migrated from catalogue into virtual directory 2.Migration Situation afterwards can correspond to Fig. 2.
In this way, there is currently two virtual directories, i.e. virtual directory 1 and virtual directory 2 in file system.With first number According to constantly be stored in, when the memory capacity that there is any virtual directory transfinites, can again perform step 104.In this way, can be with A virtual directory is set up again:Virtual directory 3, virtual directory 3 is that the 3rd be sequentially arranged in after virtual directory 2 is virtual Catalogue.According to above-mentioned formula (2), grouping strategy can be:Tactic three virtual directories are corresponding to be grouped number successively It is 9,9 and 2, or is followed successively by 8,8 and 4, or is followed successively by 7,7 and 6.
In the embodiment of the present invention, it is contemplated that improve the unbalanced degree of packet between different virtual directories as far as possible, it may be preferred to The corresponding number that is grouped of tactic three virtual directories is followed successively by 7,7 and 6.
In this way, putting in order based on each virtual directory, firstly for the first virtual directory:Virtual directory 1, it is current right That answers is grouped into packet 1~packet 10, and number is 10, and its corresponding packet number determined is 7, and both are at difference 3, therefore there are 3 packets to be migrated during virtual directory 1 can be determined:Packet 8~packet 10, therefore can be to be migrated by this 3 Packet is migrated to second virtual directory from virtual directory 1:In virtual directory 2.
Then, for second virtual directory:Virtual directory 2, due to having completed moving to packet 8~packet 10 at present Move, thus virtual directory 2 it is current it is corresponding be grouped into packet 8~packet 20, number is 13, and its corresponding packet determined Number is 7, and both differences are 6, therefore there are 6 packets to be migrated in can determining virtual directory 2:Packet 15~packet 20, therefore this 6 packets to be migrated can be migrated to the 3rd virtual directory from virtual directory 2:In virtual directory 3.
Subsequently, for the 3rd virtual directory:Virtual directory 3, due to having completed moving to packet 15~packet 20 at present Move, thus virtual directory 3 it is current it is corresponding be grouped into packet 15~packet 20, number is 6, and its corresponding packet determined Number is 6, and both differences are 0, therefore does not exist packet to be migrated in can determining virtual directory 3, therefore can terminate this Packet migration flow.
It is same as above, with being constantly stored in for metadata, when the memory capacity that there is any virtual directory in file system When transfiniting, step 104 can be again performed.In this way, a virtual directory can be set up:Virtual directory 4.Then, complete to divide After the execution of group migration, virtual directory 1 can correspond to packet 1~packet 5, and virtual directory 2 can correspond to packet 6~packet 10, virtual directory 3 can correspond to packet 11~packet 15, and virtual directory 4 can correspond to packet 16~packet 20.
So circulation, for the whole virtual directories that there is currently, as long as monitoring there is the storage of any virtual directory Capacity transfinites, and can repeat step 104, is not transfinited with the memory capacity for ensureing any virtual directory, so as to ensure The smooth execution of the operations such as retrieval, the cache management of metadata.
Based on the above, after packet migration is completed, the packet of last position virtual directory and virtual directory adjacent thereto Number may be different.For example, when there is currently three virtual directories, the corresponding packet number of three virtual directories can be successively It is 7,7 and 6.Due to when the unbalanced degree of the packet between two virtual directories of presence is higher, whole file system can be influenceed The metadata management stability of system, therefore can be adjusted with to the unbalanced degree of packet higher by way of adding packet.
Therefore, in an embodiment of the invention, in order to illustrate a kind of possibility implementation for adjusting the unbalanced degree of packet, So, further include after step 104:Monitor the last position virtual directory and virtual directory adjacent thereto point The unbalanced degree of group is not located at when in the range of Second Threshold set in advance, sets up at least corresponding to the last position virtual directory Individual packet, so that the unbalanced degree of the packet is located in the range of the Second Threshold;It is grouped for the whole that there is currently, again Determine that each is grouped corresponding mapping range;It is performed both by for the corresponding metadata of each file:Judge this yuan Whether the packet and packet corresponding with the mapping range that it matches that data are currently located are identical, if it is not, the metadata is moved In moving to packet corresponding with the mapping range that it matches.
Based on the above, for example, when there is currently three virtual directories, the corresponding packet of three virtual directories Number can be followed successively by 7,7 and 6.The first two virtual group both corresponds to 7 packets, therefore the unbalanced degree of its packet is closed Lattice, and latter two virtual group there may be packet it is unbalanced.In this way, a packet can be increased newly, and the newly-increased packet is right Ying Yumo virtual directory, so that last position virtual directory is same to that should have 7 packets, to adjust its dividing with adjacent virtual catalogue The unbalanced degree of group.
For newly-established packet, it is thus necessary to determine that its corresponding mapping range, therefore need currently to be deposited in adjustment file system Each packet mapping range.For example, there is currently 20 packets, 1~packet 20, and this 20 points are respectively grouped The corresponding mapping range of group be followed successively by [0,10], (10,20] ..., (190,200].After a newly-increased packet, there is currently 21 packets, therefore [0,200] this total mapping range can be redistributed.
For example, this 21 corresponding mapping ranges of packet can be followed successively by [0,10], (10,20] ..., (100,110], (110,119], (119,128] ..., (191,200].As can be seen that the mapping range of packet 1~packet 11 does not change, and Packet 12 to packet 20 mapping range change, such as be grouped 13 mapping range by (120,130] be changed to (119, 128], and a newly-increased order of packets is used as packet 21, can to should have mapping range (191,200].
By taking above-mentioned packet 13 as an example, it is assumed that for a certain metadata, cryptographic Hash can be calculated for 129,129 can fall into (120,130] in, therefore the metadata can be mapped to and be grouped in 13.But, when packet 13 mapping range be changed into (119, 128] still it is 129 because the cryptographic Hash for calculating is constant after, therefore can be seen that 129 can fall into corresponding to packet 14 Mapping range (128,137] in, therefore can determine that the metadata corresponds to packet 14.In this way, because the metadata is currently located at In packet 13, and determine be grouped into packet 14, both are inconsistent, therefore the metadata 13 can be migrated to packet from packet In 14.
Based on same realization principle, corresponding migration operation can be carried out to each metadata to be migrated.In this way, new The packet of foundation:Can also be stored with a number of metadata in packet 21, such that it is able to make the difference in current all packets The metadata quantity stored in packet is more or less the same, so that the memory capacity of each virtual directory can not only be alleviated, while can Reduce the unbalanced degree of packet between different virtual directories.
Certainly, for each new metadata, can be according to the current mapping range of each packet, by each new metadata Map to during correspondence is grouped and stored.
In an embodiment of the invention, the number of at least one packet of the foundation is to meet described so that described Minimum number of the unbalanced degree of packet in the range of the Second Threshold.
As an example it is assumed that virtual directory corresponding packet number in last position is 8, virtual directory adjacent thereto is corresponding Packet number is 10, and both unbalanced degree of packet are 20%, are unsatisfactory for the Second Threshold scope of setting:≤ 10%, therefore can To increase by 1 packet or 2 packets, accordingly, so that virtual directory corresponding packet number in last position is 9 or 10, so that The unbalanced degree of packet for making both is 10% or 0%, to meet Second Threshold scope:≤ 10%.In this way, due to setting up 1 or 2 It is individual, therefore minimum value can be taken, that is, set up 1 packet.
In an embodiment of the invention, in order to illustrate a kind of possibility implementation for calculating the unbalanced degree of packet, so, The unbalanced degree of packet meets following formula (3);
Wherein, Y is the unbalanced degree of packet, XnIt is the number of the corresponding packet of the last position virtual directory, XmIt is and institute State the number of the adjacent corresponding packet of virtual directory of last position virtual directory.
In detail, it is grouped that unbalanced degree is lower, illustrates the number of the corresponding packet of two virtual directories closer to more beneficial In the metadata management stability for ensureing whole file system.
In sum, the embodiment of the present invention proposes a kind of method for managing metadata, by real-time dynamic newly-built void Intend catalogue, by the corresponding different grouping of each virtual directory, to be managed to magnanimity metadata, it is to avoid all metadata are equal All the time it is stored under catalogue, such that it is able to improve the recall precision to magnanimity metadata.The memory capacity of each virtual directory transfinites When, can be solved by the implementation of newly-built virtual directory and migration packet, and be grouped between different virtual directories uneven During weighing apparatus, can by it is newly-built packet and migration metadata implementation be solved so that realize whole file system according to Practical application carries out dynamic expansion.
As shown in figure 3, the method that one embodiment of the invention provides another management metadata, specifically includes following step Suddenly:
Step 301:For corresponding 20 packets of catalogue, each corresponding mapping range of packet is determined.
In detail, tactic 20 packets are there is currently in file system, and both corresponds to catalogue, be respectively grouped 1~packet 20, and this 20 corresponding mapping ranges of packet be followed successively by [0,10], (10,20] ..., (190,200].
Step 302:It is performed both by for the corresponding metadata of each file:According to including file title, meter Calculate the corresponding cryptographic Hash of title;It is determined that the target mapping range matched with cryptographic Hash, wherein, the cryptographic Hash maps positioned at target In the range of;The metadata is stored into the corresponding packet of target mapping range.
In detail, the title of this document is included in the corresponding metadata of each file.For example, the corresponding unit of file Data can be including the title of file, time, authority, extended attribute, storage location etc..
For example, it is assumed that for a certain metadata, the cryptographic Hash for calculating is 17, above-mentioned 20 packets can be fallen into due to 17 In second packet:Be grouped 2 corresponding mapping ranges (10,20], illustrate the metadata and mapping range (10,20] phase Match somebody with somebody, therefore stored in the metadata being mapped into packet 2.
Step 303:When the memory capacity for monitoring catalogue is not located in the range of first threshold set in advance, step is performed Rapid 304.
In detail, memory capacity can be calculated and obtained by above-mentioned formula (1).
With being constantly stored in for metadata, it is assumed that be computed, the memory capacity of catalogue is 110,000, not positioned at first threshold model Enclose:In≤10 ten thousand.
Step 304:The virtual directory of the first quantity is set up, and according to predetermined packet migration rules, for current Whole packets of presence, carry out corresponding migration process.
In detail, the first quantity can be by migration process, so that the storage of any virtual directory that there is currently is held Amount is respectively positioned on the minimum number in the range of the first threshold.
Based on the above, it can be seen that, the minimum number of first quantity is 2.Therefore, it can set up 2 virtually Catalogue:Virtual directory 1 and virtual directory 2.The order of virtual directory 2 is arranged in after virtual directory 1.Therefore virtual directory 1 is first place Virtual directory, virtual directory 2 is second virtual directory.If a newly-built virtual directory, the newly-built virtual directory again Order is used as the 3rd virtual directory.So circulation.
In detail, after completing to set up each virtual directory, the corresponding metadata of the virtual directory can be generated.Its In, the corresponding metadata of each virtual directory of generation can be stored into catalogue, be managed collectively with by catalogue.
In detail, the packet migration rules can be:Each order of packets arrangement in the whole packets that there is currently, and work as When each virtual directory order in whole virtual directories of preceding presence is arranged, each virtual mesh is determined according to above-mentioned formula (2) Record the number of corresponding packet;
When the number of the whole virtual directories that there is currently is first quantity, according to each the virtual mesh determined The number of corresponding packet is recorded, described at least two packets are migrated into each virtual directory that there is currently respectively, its In, for any two virtual directory, when second virtual directory that is located therein of putting in order of the first virtual directory therein Put in order after when, the putting in order of each packet in first virtual directory is respectively positioned on each in second virtual directory Packet put in order after;
When the number of the whole virtual directories that there is currently not is first quantity, the arrangement based on each virtual directory is suitable Sequence, is performed both by for each virtual directory successively:The number for calculating the current corresponding packet of the virtual directory is subtracted and determined The corresponding packet of the virtual directory number difference;When it is positive number to judge the difference, the virtual directory pair is determined The packet to be migrated answered, wherein, in the current corresponding all packets of the virtual directory, the position that puts in order of the packet to be migrated After other packets, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order It is arranged in the next virtual directory after the virtual directory.
In embodiments of the present invention, two virtual directories be presently, there are:Virtual directory 1 and virtual directory 2.There is currently The number of whole virtual directories is 2, equal to the first quantity:2, that is, illustrate that the whole virtual directories that there is currently are virtual mesh The foundation first of record, therefore according to above-mentioned packet migration rules, packet 1~packet 10 can be migrated to virtual directory from catalogue In 1, packet 11~packet 20 is migrated into virtual directory 2 from catalogue.
Step 305:When monitoring that the memory capacity that there is any virtual directory is not located in the range of first threshold, perform Step 304, and perform step 306.
Accordingly, based on same realization principle, same to above-mentioned formula (1), in an embodiment of the invention, for any Virtual directory, the memory capacity of the virtual directory can be, the metadata stored in corresponding each packet of the virtual directory Number plus and.
For the whole virtual directories that be there is currently in file system, if the memory capacity that there is any virtual directory surpasses Limit, can be held by performing step 304, newly-built virtual directory and to carry out respective packets migration with the storage for solving to occur Amount overrun issues.
If for example, detect virtual directory 1 transfinite, therefore can be with a newly-built virtual directory:Virtual directory 3.Now, currently The number of whole virtual directories of presence is 3, is not equal to the first quantity:1, that is, illustrate the whole virtual directories that there is currently It is not the foundation first of virtual directory.
In this way, according to above-mentioned packet migration rules, packet 8~packet 10 can be migrated to virtual mesh from virtual directory 1 In record 2, packet 15~packet 20 is migrated into virtual directory 3 from virtual directory 2.Now, these three virtual directories are corresponding Packet number is respectively 7,7 and 6.
Similarly, how many virtual directories no matter are there is currently, as long as monitoring there is the memory capacity of any virtual directory Transfinite, can be by repeating step 304, newly-built virtual directory and to carry out respective packets migration, to solve what is occurred Memory capacity overrun issues.
Step 306:It is not located in the unbalanced degree of packet for monitoring last position virtual directory and virtual directory adjacent thereto When in the range of Second Threshold set in advance, at least one packet for corresponding to last position virtual directory is set up, so that packet is uneven Weighing apparatus degree is located in the range of Second Threshold.
In detail, being grouped unbalanced degree can be calculated and be obtained by above-mentioned formula (3).
In detail, the number of at least one packet of foundation can be to meet so that being grouped unbalanced degree is located at the second threshold Minimum number in the range of value.
Assuming that three virtual directories are there is currently, and the corresponding packet number of these three virtual directories is respectively 7,7 With 6.It is computed, the unbalanced degree of packet of virtual directory 2 and virtual directory 3 is 14.2%, not positioned at Second Threshold scope:≤ In 10%, therefore can be with a newly-built packet corresponding to virtual directory 3:Packet 21.
Step 307:For the whole packet that there is currently, redefine each and be grouped corresponding mapping model Enclose.
After a newly-increased packet, 21 packets are there is currently, therefore this always maps model can to redistribute [0,200] Enclose.For example, this 21 corresponding mapping ranges of packet can be followed successively by [0,10], (10,20] ..., (100,110], (110, 119], (119,128] ..., (191,200].
Step 308:It is performed both by for the corresponding metadata of each file:Judge packet that the metadata is currently located and Whether packet corresponding with the mapping range that it matches is identical, if so, terminate current process, otherwise, by the metadata migration Into packet corresponding with the mapping range that it matches.
To be grouped as a example by 13, it is assumed that for a certain metadata, it is 129 that can calculate cryptographic Hash, illustrates that it there is currently In packet 13.But, when packet 13 mapping range be changed into (119,128] after, because the cryptographic Hash for calculating is constant, still Be 129, thus can be seen that 129 can fall into corresponding to packet 14 mapping range (128,137] in.In this way, can be by this yuan Data are migrated into packet 14 from packet 13.
Based on same realization principle, corresponding migration operation can be carried out to each metadata to be migrated.
Certainly, for each new metadata, can be according to the current mapping range of each packet, by each new metadata Map to during correspondence is grouped and stored.
As shown in figure 4, one embodiment of the invention provides a kind of device for managing metadata, including:
Determining unit 401, for for corresponding at least two packet of catalogue, determining that each described packet corresponding is reflected Penetrate scope;
Map unit 402, for being performed both by for the corresponding metadata of each file:By the metadata store to its In the corresponding packet of mapping range for matching, and trigger the first monitoring unit 403;
First monitoring unit 403, for monitoring that the memory capacity of the catalogue is not located at set in advance the When in one threshold range, processing unit 404 is triggered;
The processing unit 404, the virtual directory for setting up the first quantity, and according to predetermined packet migration rule Then, for the whole packet that there is currently, corresponding migration process is carried out.
In an embodiment of the invention, the title of this document is included in the corresponding metadata of each file;
The map unit 402, specifically for being performed both by for the corresponding metadata of each file:According to including File title, calculate the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, its In, the cryptographic Hash is located in the target mapping range;The metadata is stored to corresponding point of the target mapping range In group.
In an embodiment of the invention, the memory capacity of the catalogue meets above-mentioned formula (1).
In an embodiment of the invention, first quantity is by the migration process, so that what be there is currently appoints The memory capacity of one virtual directory is respectively positioned on the minimum number in the range of the first threshold.
In an embodiment of the invention, first monitoring unit 403, is additionally operable to monitoring there is any virtual mesh The memory capacity of record is not located at when in the range of the first threshold, triggers the processing unit 404.
In an embodiment of the invention, the packet migration rules, including:Each point in the whole packets that there is currently Group order arrange, and in the whole virtual directories that there is currently each virtual directory order arrange when, according to above-mentioned formula (2) really The number of the corresponding packet of fixed each virtual directory;
When the number of the whole virtual directories that there is currently is first quantity, according to each the virtual mesh determined The number of corresponding packet is recorded, described at least two packets are migrated into each virtual directory that there is currently respectively, its In, for any two virtual directory, when second virtual directory that is located therein of putting in order of the first virtual directory therein Put in order after when, the putting in order of each packet in first virtual directory is respectively positioned on each in second virtual directory Packet put in order after;
When the number of the whole virtual directories that there is currently not is first quantity, the arrangement based on each virtual directory is suitable Sequence, is performed both by for each virtual directory successively:The number for calculating the current corresponding packet of the virtual directory is subtracted and determined The corresponding packet of the virtual directory number difference;When it is positive number to judge the difference, the virtual directory pair is determined The packet to be migrated answered, wherein, in the current corresponding all packets of the virtual directory, the position that puts in order of the packet to be migrated After other packets, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order It is arranged in the next virtual directory after the virtual directory.
In an embodiment of the invention, Fig. 5 is refer to, the device of the management metadata can also include:Second monitoring Unit 502 is set up in unit 501, packet;
Second monitoring unit 501, for monitoring the last position virtual directory and virtual directory adjacent thereto Packet it is unbalanced degree be located at Second Threshold set in advance in the range of when, trigger it is described packet set up unit 502;
Unit 502 is set up in the packet, for setting up at least one packet corresponding to the last position virtual directory, so that The unbalanced degree of packet is located in the range of the Second Threshold, and triggers the determining unit 401;
The determining unit 401, be additionally operable to receive it is described packet set up the trigger signal that unit 502 is sent when, pin Whole packet to there is currently, redefines each and is grouped corresponding mapping range, and it is single to trigger the mapping Unit 402;
The map unit 402, is additionally operable to when the trigger signal that the determining unit 401 is sent is received, for every The corresponding metadata of one file is performed both by:Judge the packet that the metadata is currently located and the mapping range pair matched with it Whether the packet answered is identical, if it is not, by the metadata migration to packet corresponding with the mapping range that it matches.
In an embodiment of the invention, the number that at least one packet of the foundation of unit 502 is set up in the packet is to accord with Close described so that minimum number of the unbalanced degree of packet in the range of the Second Threshold.
In an embodiment of the invention, the unbalanced degree of the packet meets above-mentioned formula (3).
The contents such as the information exchange between each unit, implementation procedure in said apparatus, due to implementing with the inventive method Example is based on same design, and particular content can be found in the narration in the inventive method embodiment, and here is omitted.
In sum, each embodiment of the invention at least has the advantages that:
1st, in the embodiment of the present invention, for corresponding at least two packet of catalogue, the corresponding mapping range of each packet is determined; The corresponding metadata of each file is stored into packet corresponding with the mapping range that it matches respectively;Monitoring mesh The memory capacity of record is not located at when in preset threshold range, sets up the virtual directory of the first quantity, and according to packet migration rules, Corresponding migration process is carried out for the whole packets that there is currently.Based on packet to store magnanimity metadata respectively, therefore in catalogue Memory capacity when transfiniting, can be deposited with by the way that virtual directory is grouped by way of increasing virtual directory and migrating packet Storage metadata, such that it is able to solve the problems, such as that catalogue memory capacity transfinites.Therefore, the embodiment of the present invention can improve metadata Recall precision.
2nd, in the embodiment of the present invention, it is proposed that a kind of method for managing metadata, by real-time dynamic newly-built virtual mesh Record, by the corresponding different grouping of each virtual directory, to be managed to magnanimity metadata, it is to avoid by all metadata all the time It is stored under catalogue, such that it is able to improve the recall precision to magnanimity metadata.When the memory capacity of each virtual directory transfinites, can When being solved with the implementation by newly-built virtual directory and migration packet, and being grouped unbalanced between different virtual directories, Can be solved by the implementation of newly-built packet and migration metadata, so as to realize that whole file system is answered according to actual With carrying out dynamic expansion.
It should be noted that herein, such as first and second etc relational terms are used merely to an entity Or operation makes a distinction with another entity or operation, and not necessarily require or imply these entities or exist between operating Any this actual relation or order.And, term " including ", "comprising" or its any other variant be intended to it is non- It is exclusive to include, so that process, method, article or equipment including a series of key elements not only include those key elements, But also other key elements including being not expressly set out, or also include by this process, method, article or equipment are solid Some key elements.In the absence of more restrictions, the key element limited by sentence " including ", does not arrange Except also there is other identical factor in the process including the key element, method, article or equipment.
One of ordinary skill in the art will appreciate that:Realizing all or part of step of above method embodiment can pass through Programmed instruction related hardware is completed, and foregoing program can be stored in the storage medium of embodied on computer readable, the program Upon execution, the step of including above method embodiment is performed;And foregoing storage medium includes:ROM, RAM, magnetic disc or light Disk etc. is various can be with the medium of store program codes.
It is last it should be noted that:Presently preferred embodiments of the present invention is the foregoing is only, skill of the invention is merely to illustrate Art scheme, is not intended to limit the scope of the present invention.All any modifications made within the spirit and principles in the present invention, Equivalent, improvement etc., are all contained in protection scope of the present invention.

Claims (10)

1. it is a kind of manage metadata method, it is characterised in that including:
S1:For corresponding at least two packet of catalogue, each corresponding mapping range of the packet is determined;
S2:It is performed both by for the corresponding metadata of each file:The metadata is stored to the mapping range matched with it In corresponding packet;
S3:When the memory capacity for monitoring the catalogue is not located in the range of first threshold set in advance, S4 is performed;
S4:The virtual directory of the first quantity is set up, and according to predetermined packet migration rules, for the whole that there is currently Packet, carries out corresponding migration process.
2. method according to claim 1, it is characterised in that
The title of this document is included in the corresponding metadata of each file;
The S2, including:It is performed both by for the corresponding metadata of each file:According to including file title, meter Calculate the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, wherein, the cryptographic Hash position In in the target mapping range;The metadata is stored into the corresponding packet of the target mapping range;
And/or,
The memory capacity coincidence formula one of the catalogue, wherein,
The formula one includes:
X = Σ i = 1 n N i
Wherein, X is the memory capacity of the catalogue, and n is the number of at least two packet, NiIt is described at least two packets In i-th packet in store metadata number;
And/or,
First quantity is by the migration process, so that the memory capacity of any virtual directory that there is currently is respectively positioned on Minimum number in the range of the first threshold;
And/or,
Further included after S4:In the memory capacity for monitoring to have any virtual directory not positioned at the first threshold model When enclosing interior, S4 is performed.
3. method according to claim 1 and 2, it is characterised in that
The packet migration rules, including:Each order of packets arrangement in the whole packets that there is currently, and there is currently it is complete When each virtual directory order in portion's virtual directory is arranged, the individual of the corresponding packet of each virtual directory is determined according to formula two Number;
When the number of the whole virtual directories that there is currently is first quantity, according to each virtual directory pair determined The number of the packet answered, described at least two packets are migrated into each virtual directory that there is currently respectively, wherein, it is right In any two virtual directory, when the arrangement of second virtual directory being located therein that puts in order of the first virtual directory therein When after order, each packet that the putting in order of each packet in first virtual directory is respectively positioned in second virtual directory After putting in order;
When the number of the whole virtual directories that there is currently not is first quantity, based on putting in order for each virtual directory, It is performed both by for each virtual directory successively:Calculate the current corresponding packet of the virtual directory number subtract determine should The difference of the number of the corresponding packet of virtual directory;When it is positive number to judge the difference, determine that the virtual directory is corresponding Packet to be migrated, wherein, in the current corresponding all packets of the virtual directory, the packet to be migrated puts in order positioned at it After he is grouped, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order and is arranged In next virtual directory after the virtual directory;
The formula two includes:
X = ( n - 1 ) × x 1 + x 2 x 1 ≥ x 2
Wherein, X is the number of the whole packets that there is currently;N is the number of the whole virtual directories that there is currently;x1For current In whole virtual directories of presence, the number of the corresponding packet of any virtual directory in addition to last position virtual directory, x2For described The number of the corresponding packet of last position virtual directory.
4. method according to claim 3, it is characterised in that
Further included after S4:It is uneven in the packet for monitoring the last position virtual directory and virtual directory adjacent thereto Weighing apparatus degree is not located at when in the range of Second Threshold set in advance, sets up at least one point corresponding to the last position virtual directory Group, so that the unbalanced degree of the packet is located in the range of the Second Threshold;For the whole packet that there is currently, redefine Each is grouped corresponding mapping range;It is performed both by for the corresponding metadata of each file:Judge the metadata Whether the packet and packet corresponding with the mapping range that it matches being currently located are identical, if it is not, by the metadata migration extremely In packet corresponding with the mapping range that it matches.
5. method according to claim 4, it is characterised in that
The number of at least one packet of the foundation is to meet described so that the unbalanced degree of packet is located at second threshold Minimum number in the range of value;
And/or,
The unbalanced degree coincidence formula three of packet, wherein,
The formula three includes:
Y = X m - X n X m × 100 %
Wherein, Y is the unbalanced degree of packet, XnIt is the number of the corresponding packet of the last position virtual directory, XmIt is and the end The number of the adjacent corresponding packet of virtual directory of position virtual directory.
6. it is a kind of manage metadata device, it is characterised in that including:
Determining unit, for for corresponding at least two packet of catalogue, determining each corresponding mapping range of the packet;
Map unit, for being performed both by for the corresponding metadata of each file:The metadata is stored to it and is matched The corresponding packet of mapping range in, and trigger the first monitoring unit;
First monitoring unit, for not being located at first threshold model set in advance in the memory capacity for monitoring the catalogue When enclosing interior, processing unit is triggered;
The processing unit, the virtual directory for setting up the first quantity, and according to predetermined packet migration rules, for The whole packets that there is currently, carry out corresponding migration process.
7. it is according to claim 6 management metadata device, it is characterised in that
The title of this document is included in the corresponding metadata of each file;
The map unit, specifically for being performed both by for the corresponding metadata of each file:According to including file Title, calculate the corresponding cryptographic Hash of the title;It is determined that the target mapping range matched with the cryptographic Hash, wherein, institute Cryptographic Hash is stated to be located in the target mapping range;The metadata is stored into the corresponding packet of the target mapping range;
And/or,
The memory capacity coincidence formula one of the catalogue, wherein,
The formula one includes:
X = Σ i = 1 n N i
Wherein, X is the memory capacity of the catalogue, and n is the number of at least two packet, NiIt is described at least two packets In i-th packet in store metadata number;
And/or,
First quantity is by the migration process, so that the memory capacity of any virtual directory that there is currently is respectively positioned on Minimum number in the range of the first threshold;
And/or,
First monitoring unit, is additionally operable in the memory capacity for monitoring to have any virtual directory not positioned at first threshold When in the range of value, the processing unit is triggered.
8. the device of the management metadata according to claim 6 or 7, it is characterised in that
The packet migration rules, including:Each order of packets arrangement in the whole packets that there is currently, and there is currently it is complete When each virtual directory order in portion's virtual directory is arranged, the individual of the corresponding packet of each virtual directory is determined according to formula two Number;
When the number of the whole virtual directories that there is currently is first quantity, according to each virtual directory pair determined The number of the packet answered, described at least two packets are migrated into each virtual directory that there is currently respectively, wherein, it is right In any two virtual directory, when the arrangement of second virtual directory being located therein that puts in order of the first virtual directory therein When after order, each packet that the putting in order of each packet in first virtual directory is respectively positioned in second virtual directory After putting in order;
When the number of the whole virtual directories that there is currently not is first quantity, based on putting in order for each virtual directory, It is performed both by for each virtual directory successively:Calculate the current corresponding packet of the virtual directory number subtract determine should The difference of the number of the corresponding packet of virtual directory;When it is positive number to judge the difference, determine that the virtual directory is corresponding Packet to be migrated, wherein, in the current corresponding all packets of the virtual directory, the packet to be migrated puts in order positioned at it After he is grouped, and the number of the packet to be migrated is equal to the difference;The packet to be migrated is migrated to order and is arranged In next virtual directory after the virtual directory;
The formula two includes:
X = ( n - 1 ) × x 1 + x 2 x 1 ≥ x 2
Wherein, X is the number of the whole packets that there is currently;N is the number of the whole virtual directories that there is currently;x1For current In whole virtual directories of presence, the number of the corresponding packet of any virtual directory in addition to last position virtual directory, x2For described The number of the corresponding packet of last position virtual directory.
9. it is according to claim 8 management metadata device, it is characterised in that
Also include:Unit is set up in second monitoring unit, packet;
Second monitoring unit, in the packet for monitoring the last position virtual directory and virtual directory adjacent thereto not Equilibrium degree is not located at when in the range of Second Threshold set in advance, is triggered the packet and is set up unit;
Unit is set up in the packet, for setting up at least one packet corresponding to the last position virtual directory, so that described point The unbalanced degree of group is located in the range of the Second Threshold, and triggers the determining unit;
The determining unit, is additionally operable to when receiving the packet and setting up the trigger signal that unit is sent, for there is currently Whole packets, redefine each and be grouped corresponding mapping range, and trigger the map unit;
The map unit, is additionally operable to when the trigger signal that the determining unit is sent is received, for each file pair The metadata answered is performed both by:Judging packet that the metadata is currently located and packet corresponding with the mapping range that it matches is It is no identical, if it is not, by the metadata migration to packet corresponding with the mapping range that it matches.
10. it is according to claim 9 management metadata device, it is characterised in that
The number that at least one packet of unit foundation is set up in the packet is to meet described so that the unbalanced degree position of packet Minimum number in the range of the Second Threshold;
And/or,
The unbalanced degree coincidence formula three of packet, wherein,
The formula three includes:
Y = X m - X n X m × 100 %
Wherein, Y is the unbalanced degree of packet, XnIt is the number of the corresponding packet of the last position virtual directory, XmIt is and the end The number of the adjacent corresponding packet of virtual directory of position virtual directory.
CN201611139129.6A 2016-12-12 2016-12-12 Method and device for managing metadata Active CN106777062B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201611139129.6A CN106777062B (en) 2016-12-12 2016-12-12 Method and device for managing metadata

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201611139129.6A CN106777062B (en) 2016-12-12 2016-12-12 Method and device for managing metadata

Publications (2)

Publication Number Publication Date
CN106777062A true CN106777062A (en) 2017-05-31
CN106777062B CN106777062B (en) 2020-03-10

Family

ID=58879884

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201611139129.6A Active CN106777062B (en) 2016-12-12 2016-12-12 Method and device for managing metadata

Country Status (1)

Country Link
CN (1) CN106777062B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102497428A (en) * 2011-12-13 2012-06-13 方正国际软件有限公司 Remote storage system and method for remote storage thereof
CN103688257A (en) * 2012-11-27 2014-03-26 华为技术有限公司 Method and device for managing metadata
CN103916459A (en) * 2014-03-04 2014-07-09 南京邮电大学 Big data filing and storing system
CN103916467A (en) * 2014-03-25 2014-07-09 中国科学院计算技术研究所 Load transfer method and system in metadata cluster
CN106021462A (en) * 2016-05-17 2016-10-12 深圳市中博科创信息技术有限公司 File storage method of cluster file system and cluster file system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102497428A (en) * 2011-12-13 2012-06-13 方正国际软件有限公司 Remote storage system and method for remote storage thereof
CN103688257A (en) * 2012-11-27 2014-03-26 华为技术有限公司 Method and device for managing metadata
CN103916459A (en) * 2014-03-04 2014-07-09 南京邮电大学 Big data filing and storing system
CN103916467A (en) * 2014-03-25 2014-07-09 中国科学院计算技术研究所 Load transfer method and system in metadata cluster
CN106021462A (en) * 2016-05-17 2016-10-12 深圳市中博科创信息技术有限公司 File storage method of cluster file system and cluster file system

Also Published As

Publication number Publication date
CN106777062B (en) 2020-03-10

Similar Documents

Publication Publication Date Title
US10176113B2 (en) Scalable indexing
CA2131968C (en) Computer method and system for b-tree optimization
CN107423422B (en) Spatial data distributed storage and search method and system based on grid
US9336152B1 (en) Method and system for determining FIFO cache size
CN103581331B (en) The online moving method of virtual machine and system
CN103995855B (en) The method and apparatus of data storage
CN106612619B (en) Hierarchical stor, using the computer and correction of hierachical storing device to the computer of the access count of file
CN103140840B (en) The method and device of data management
CN106911743B (en) Small documents write polymerization, read polymerization and system and client
CN101419616A (en) Data synchronization method and apparatus
CN107273042A (en) Deduplication DRAM system algorithm framework
CN103701469B (en) A kind of compression and storage method of large-scale graph data
CN107315814A (en) Data consistency checking method and system after a kind of KDB database data migrations
CN107958079A (en) Aggregate file delet method, system, device and readable storage medium storing program for executing
CN107870970A (en) A kind of data store query method and system
CN104346384A (en) Method and device for processing small files
CN110597835B (en) Transaction data deleting method and device based on blockchain
CN104021223B (en) The access method and device of a kind of Cluster Database measuring point
CN107391045A (en) A kind of directory space quota method and system
CN102722450B (en) Storage method for redundancy deletion block device based on location-sensitive hash
CN113867627B (en) Storage system performance optimization method and system
US9275091B2 (en) Database management device and database management method
CN106777062A (en) A kind of method and device for managing metadata
CN106933974A (en) The structure and querying method and device of a kind of video file index information
CN107273443A (en) A kind of hybrid index method based on big data model metadata

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant