CN107766159A - A kind of metadata management method, device and computer-readable recording medium - Google Patents
A kind of metadata management method, device and computer-readable recording medium Download PDFInfo
- Publication number
- CN107766159A CN107766159A CN201711116398.5A CN201711116398A CN107766159A CN 107766159 A CN107766159 A CN 107766159A CN 201711116398 A CN201711116398 A CN 201711116398A CN 107766159 A CN107766159 A CN 107766159A
- Authority
- CN
- China
- Prior art keywords
- meta data
- data server
- load value
- value
- underloading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a kind of metadata management method, device and computer-readable recording medium, each self-corresponding load value of all meta data servers in group system and hot value are obtained；According to load value corresponding to predetermined threshold value and each meta data server, overload meta data server and underloading meta data server are selected；According to load value corresponding to all meta data servers, mean load value is calculated；According to load value and hot value corresponding to load value and hot value corresponding to mean load value, predetermined threshold value, overload meta data server and each underloading meta data server, the catalogue subtree for overloading respective numbers in meta data server is migrated to the target metadata server selected.In this process, the temperature situation of each meta data server has been taken into full account, so as to more reasonably enter the migration of column catalogue subtree, the number of catalogue subtree migration has been effectively reduced, improves the efficiency for realizing load balancing.
Description
Technical field
The present invention relates to storage cluster technical field, more particularly to a kind of metadata management method, device and computer
Readable storage medium storing program for executing.
Background technology
Distributed file system (distributed file system, CEPH) is by by the resource tissue of more machines
Get up, unified, Large Copacity, high performance, highly reliable file service is externally provided, meets wanting for large-scale application
Ask.
CEPH is generally made up of meta data server (MetaDataServer, MDS) cluster and storage server cluster.System
Meter shows, in the access of file system, the 50-80% of whole access times is accounted for the access times of metadata.It is big to tackle
The metadata operation request of amount, ensures good performance and autgmentability, the way to manage of metadata and the load balancing of MDS clusters
Strategy is of crucial importance.
In traditional approach, the ability that migration algorithm has given tacit consent to each MDS is identical, and actually same metadata access is not
It is different with load caused by MDS.Catalogue subtree corresponding to metadata is moved to from the strong MDS of high capacity, ability
When on the weak MDS of low-load, ability, the weak MDS of ability may receive excessive metadata load, be quickly become new system bottle
Neck；Although in the migratory movement of a new round, catalogue subtree can migrate out from the weak MDS of ability, but still waste a large amount of
Network traffics, extend time of load balancing.And when catalogue subtree moves to low bear from the weak MDS of high capacity, ability
When on the strong machine of load, ability, although the waste of network traffics will not be caused, more temperature mistakes can essentially be migrated
Go, do not make full use of the chance of migration.
It is those skilled in the art's urgent problem to be solved it can be seen that how to lift the efficiency of load balancing.
The content of the invention
The purpose of the embodiment of the present invention is to provide a kind of metadata management method, device and computer-readable recording medium,
Can be with.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of metadata management method, including：
Obtain each self-corresponding load value of all meta data servers and hot value in group system；
According to load value corresponding to predetermined threshold value and each meta data server, overload Metadata Service is selected
Device and underloading meta data server；
According to all load values, mean load value corresponding to the group system is calculated；
According to load value and heat corresponding to the mean load value, the predetermined threshold value, the overload meta data server
Load value and hot value corresponding to angle value and each underloading meta data server, from all underloading Metadata Services
Select target metadata server in device, and by the catalogue subtree of respective numbers in the overload meta data server migrate to
The target metadata server.
Optionally, it is described according to corresponding to the mean load value, the predetermined threshold value, the overload meta data server
Load value and hot value corresponding to load value and hot value and each underloading meta data server, from all underloadings
Select target metadata server in meta data server, and by the catalogue of respective numbers in the overload meta data server
Subtree, which is migrated to the target metadata server, to be included：
According to load value corresponding to the mean load value and the overload meta data server, the overload member is calculated
Overload quantity corresponding to data server；
According to load value and heat corresponding to the mean load value, the predetermined threshold value, the overload meta data server
Load value and hot value corresponding to angle value and each underloading meta data server, calculate each underloading metadata
Bearing capacity corresponding to server；
According to bearing capacity corresponding to each underloading meta data server, from all underloading meta data servers
A maximum underloading meta data server of bearing capacity is selected as target metadata server；
According to bearing capacity corresponding to the overload quantity and the target metadata server, by the overload Metadata Service
The catalogue subtree of respective numbers is migrated to the target metadata server in device.
Optionally, it is described according to corresponding to the mean load value, the predetermined threshold value, the overload meta data server
Load value and hot value corresponding to load value and hot value and each underloading meta data server, calculate each described
Bearing capacity corresponding to underloading meta data server includes：
According to equation below, jth platform underloading meta data server pair in all underloading meta data servers is calculated
The bearing capacity mL answered；
Wherein, L represents the predetermined threshold value, and L1 represents the mean load value, PjRepresent jth platform underloading Metadata Service
Hot value corresponding to device, PiRepresent hot value corresponding to the overload meta data server, LjRepresent jth platform underloading metadata clothes
Load value corresponding to business device, LiRepresent load value corresponding to the overload meta data server.
Optionally, according to all load values, gone back before calculating mean load value corresponding to the group system
Including：
According to all each self-corresponding transmission cost values of meta data server in the group system obtained, by institute
There is the meta data server to carry out subarea processing；Wherein, at least one Metadata Service is included in each region
Device；
Accordingly, it is described according to all load values, calculate the group system corresponding to mean load value include：
Load value corresponding to the meta data server included according to each region, calculate each region pair
The mean load value answered；
It is described according to load value corresponding to the mean load value and each meta data server, select overload
Meta data server and all underloading meta data servers include：
Corresponding to each meta data server included according to the mean load value and each region
Load value, select overload meta data server and all underloading meta data servers corresponding to each region.
Optionally, in addition to：
Obtain all each self-corresponding hot values of metadata in the meta data server；
Judge whether that hot value exceedes the metadata of preset heat threshold value；
If so, catalogue subtree corresponding to the metadata is then copied in the group system to the member for meeting preparatory condition
In data server.
The embodiment of the present invention additionally provides a kind of meta data management device, including acquiring unit, selection unit, computing unit
And migration units；
The acquiring unit, for obtaining each self-corresponding load value of all meta data servers in group system and temperature
Value；
The selection unit, for according to load value corresponding to predetermined threshold value and each meta data server, choosing
Take out overload meta data server and underloading meta data server；
The computing unit, for according to all load values, calculating average load corresponding to the group system
Value；
The migration units, for according to the mean load value, the predetermined threshold value, the overload meta data server
Load value and hot value corresponding to corresponding load value and hot value and each underloading meta data server, from all institutes
State and target metadata server is selected in underloading meta data server, and respective numbers in meta data server are overloaded by described
Catalogue subtree migrate to the target metadata server.
Optionally, the migration units include the first computation subunit, the second computation subunit, choose subelement and execution
Subelement；
First computation subunit, for according to corresponding to the mean load value and the overload meta data server
Load value, calculate overload quantity corresponding to the overload meta data server；
Second computation subunit, for according to the mean load value, the predetermined threshold value, the overload metadata
Load value and hot value corresponding to load value corresponding to server and hot value and each underloading meta data server, meter
Calculate bearing capacity corresponding to each underloading meta data server；
The selection subelement, for according to bearing capacity corresponding to each underloading meta data server, from all institutes
State and a maximum underloading meta data server of bearing capacity is selected in underloading meta data server as target metadata service
Device；
The execution subelement, for according to bearing capacity corresponding to the overload quantity and the target metadata server,
The catalogue subtree of respective numbers in the overload meta data server is migrated to the target metadata server.
Optionally, second computation subunit is specifically used for according to equation below, calculates all underloading member numbers
According to bearing capacity mL corresponding to jth platform underloading meta data server in server；
Wherein, L represents the predetermined threshold value, and L1 represents the mean load value, PjRepresent jth platform underloading Metadata Service
Hot value corresponding to device, PiRepresent hot value corresponding to the overload meta data server, LjRepresent jth platform underloading metadata clothes
Load value corresponding to business device, LiRepresent load value corresponding to the overload meta data server.
Optionally, in addition to multidomain treat-ment unit；The multidomain treat-ment unit, for according to the group system obtained
In all each self-corresponding transmission cost values of meta data server, all meta data servers are carried out at subregions
Reason；Wherein, at least one meta data server is included in each region；
Accordingly, the computing unit is specifically used for corresponding to the meta data server that includes according to each region
Load value, calculate mean load value corresponding to each region；
The each institute for choosing unit and being specifically used for including according to the mean load value and each region
Load value corresponding to meta data server is stated, selects overload meta data server and all underloadings corresponding to each region
Meta data server.
Optionally, in addition to judging unit and copied cells；
The acquiring unit is additionally operable to obtain each self-corresponding heat of all metadata that the meta data server includes
Angle value；
The judging unit, for judging whether that hot value exceedes the metadata of preset heat threshold value；
If so, the copied cells is then triggered, the copied cells, for catalogue subtree corresponding to the metadata to be answered
Make in the group system and meet in the meta data server of preparatory condition.
The embodiment of the present invention additionally provides a kind of meta data management device, including：
Memory, for storing computer program；
Processor, for performing the computer program to realize such as the step of above-mentioned metadata management method.
The embodiment of the present invention additionally provides a kind of computer-readable recording medium, is deposited on the computer-readable recording medium
Computer program is contained, is realized when the computer program is executed by processor such as the step of above-mentioned metadata management method.
All each self-corresponding load values of meta data server in group system are obtained it can be seen from above-mentioned technical proposal
And hot value；According to load value corresponding to predetermined threshold value and each meta data server, overload meta data server is selected
With underloading meta data server；According to load value corresponding to all meta data servers, can calculate corresponding to group system
One mean load value；According to load value and hot value corresponding to mean load value, predetermined threshold value, overload meta data server with
And load value and hot value corresponding to each underloading meta data server, select one from all underloading meta data servers
The underloading meta data server of catalogue subtree migration is most appropriate for as target metadata server, and will overload metadata clothes
The catalogue subtree of respective numbers is migrated to the target metadata server in business device.In this process, each first number has been taken into full account
According to the temperature situation of server, so as to more reasonably enter the migration of column catalogue subtree, catalogue subtree is effectively reduced
The number of migration, improve the efficiency for realizing load balancing.
Brief description of the drawings
In order to illustrate the embodiments of the present invention more clearly, the required accompanying drawing used in embodiment will be done simply below
Introduce, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ordinary skill people
For member, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is a kind of flow chart of metadata management method provided in an embodiment of the present invention；
Fig. 2 is a kind of side for choosing target metadata server and entering the migration of column catalogue subtree provided in an embodiment of the present invention
The flow chart of method；
Fig. 3 is a kind of structural representation of meta data management device provided in an embodiment of the present invention；
Fig. 4 is a kind of hardware architecture diagram of meta data management device provided in an embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.Based on this
Embodiment in invention, for those of ordinary skill in the art under the premise of creative work is not made, what is obtained is every other
Embodiment, belong to the scope of the present invention.
In order that those skilled in the art more fully understand the present invention program, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.
Next, a kind of metadata management method that the embodiment of the present invention is provided is discussed in detail.Fig. 1 is implemented for the present invention
A kind of flow chart for metadata management method that example provides, this method include：
S101：Obtain each self-corresponding load value of all meta data servers and hot value in group system.
Multiple (more) meta data servers are usually contained in group system, every meta data server has correspondingly
Load value and hot value.
Every meta data server can regard a node as.Removed in view of node load has outside the Pass with CPU, also and interior
It is relevant to deposit occupancy, bandwidth occupancy etc., so, in embodiments of the present invention, according to the characteristic of CEPH meta data servers, it can adopt
With CPU, the weighted average of internal memory and bandwidth occupancy situation is as load value corresponding to the meta data server.With a first number
Exemplified by server, A, B, C are respectively the occupancy situation of CPU, internal memory and bandwidth, and a, b, c are corresponding weight coefficient, then this yuan
The load value of data server is equal to a*A+b*B+c*C.
The number that hot value accesses metadata with user is proportionate.Meta data server can be used for multiple metadata
It is managed, each metadata has its corresponding hot value, in embodiments of the present invention, can be by the hot value of these metadata
The mode being averaged, a hot value corresponding to the meta data server is calculated.
S102：According to load value corresponding to predetermined threshold value and each meta data server, the first number of overload is selected
According to server and underloading meta data server.
Predetermined threshold value can be used to indicate that the higher limit of meta data server loading condition.In embodiments of the present invention, may be used
It is attributed to overload meta data server so that load value to be more than or equal to the meta data server of predetermined threshold value, can be small by load value
Underloading meta data server is attributed in the meta data server of predetermined threshold value.
Each corresponding catalogue subtree of metadata, can be by respective numbers for the meta data server of load imbalance
Catalogue subtree migrated, to realize the load balancing of group system.Overload meta data server can be used to indicate that needs
Enter the meta data server of column catalogue subtree migration；Underloading meta data server, which can be used to indicate that, can receive new catalogue
The meta data server of tree.
S103：According to all load values, mean load value corresponding to the group system is calculated.
Mean load value can reflect the load distribution situation of whole group system.
S104：According to load value corresponding to the mean load value, the predetermined threshold value, the overload meta data server
With hot value and each underloading meta data server corresponding to load value and hot value, from all underloading metadata
Target metadata server is selected in server, and the catalogue subtree of respective numbers in the overload meta data server is moved
Move to the target metadata server.
Load value in view of every meta data server is a variate-value, with each metadata in meta data server
Disposition, the load value of the meta data server can occur to change accordingly, in order to ensure stable effective to realize load
Equilibrium, in the specific implementation, can be using the load value situation of change of each meta data server in a period of time as foundation, when a certain
The load value of platform overload meta data server continues for some time the migration stream higher than mean load value, then trigger catalog subtree
Journey.
For convenience of introduction, in the embodiment of the present invention by taking the migration of a catalogue subtree for overloading meta data server as an example
Expansion is introduced, and is needed the overload meta data server into the migration of column catalogue subtree for other, is referred to this metadata clothes
The handling process of business device, will not be repeated here.
Needed by one into exemplified by the overload meta data server of column catalogue subtree migration, its corresponding underloading metadata clothes
Business device can have it is multiple, when entering the migration of column catalogue subtree, it is necessary to select one from this multiple underloading meta data server
Individual underloading meta data server the most suitable overloads this corresponding in meta data server as target metadata server
Catalogue subtree migrate to target metadata server, so as to realize the load balancing of meta data server.
Next the migration for entering column catalogue subtree to how to choose target metadata server is deployed to introduce, such as Fig. 2 institutes
Show, the step includes：
S201：According to load value corresponding to the mean load value and the overload meta data server, calculate described
Overload overload quantity corresponding to meta data server.
Needed by one into exemplified by the overload meta data server of column catalogue subtree migration, overload quantity can be used to indicate that this
Overload meta data server needs the quantity of the catalogue subtree migrated.
In the specific implementation, can using this overload meta data server load value and mean load value difference as
Overload quantity.
For example, the load capacity of an overload meta data server is Li, the average value of group system is L1, then this is overloaded
Overload quantity corresponding to meta data server is Lout=Li-L1。
S202：According to load value corresponding to the mean load value, the predetermined threshold value, the overload meta data server
With hot value and each underloading meta data server corresponding to load value and hot value, calculate each underloading member
Bearing capacity corresponding to data server.
Bearing capacity can be used to indicate that the quantity for the catalogue subtree that meta data server can receive.
In view of a meta data server disposal ability except with load value mutually outside the Pass, it is also related to its hot value.
So in embodiments of the present invention, for bearing capacity corresponding to more accurately calculating every underloading meta data server, remove
Outside the load value for considering meta data server, hot value corresponding to meta data server has also been taken into full account.
Specifically, jth platform underloading member number in all underloading meta data servers can be calculated according to equation below
According to bearing capacity mL corresponding to server；
Wherein, L represents the predetermined threshold value, and L1 represents the mean load value, PjRepresent jth platform underloading Metadata Service
Hot value corresponding to device, PiRepresent hot value corresponding to the overload meta data server, LjRepresent jth platform underloading metadata clothes
Load value corresponding to business device, LiRepresent load value corresponding to the overload meta data server.
S203：According to bearing capacity corresponding to each underloading meta data server, taken from all underloading metadata
A maximum underloading meta data server of bearing capacity is selected in business device as target metadata server.
The bearing capacity of underloading meta data server is bigger, illustrates catalogue that this underloading meta data server can receive
The quantity of tree is more, can be using a maximum underloading meta data server of bearing capacity as mesh in this embodiment of the present invention
Mark meta data server.
S204：According to bearing capacity corresponding to the overload quantity and the target metadata server, by the first number of overload
Migrated according to the catalogue subtree of respective numbers in server to the target metadata server.
In embodiments of the present invention, the smaller that can be chosen in both overload quantity and bearing capacity moves as catalogue subtree
The quantity of shifting.
There are its corresponding hot value, and each first number in meta data server in view of each meta data server
According to also there is its corresponding hot value, in the specific implementation, the catalogue subtree of corresponding temperature in meta data server will can be overloaded
Migrate to target metadata server.
The migration of a subdirectory subtree is often carried out, load value corresponding to meta data server can occur to change accordingly, lead to
Cross and repeat aforesaid operations, it is possible to achieve the load balancing of group system.
All each self-corresponding load values of meta data server in group system are obtained it can be seen from above-mentioned technical proposal
And hot value；According to load value corresponding to predetermined threshold value and each meta data server, overload meta data server is selected
With underloading meta data server；According to load value corresponding to all meta data servers, can calculate corresponding to group system
One mean load value；According to load value and hot value corresponding to mean load value, predetermined threshold value, overload meta data server with
And load value and hot value corresponding to each underloading meta data server, select one from all underloading meta data servers
The underloading meta data server of catalogue subtree migration is most appropriate for as target metadata server, and will overload metadata clothes
The catalogue subtree of respective numbers is migrated to the target metadata server in business device.In this process, each first number has been taken into full account
According to the temperature situation of server, so as to more reasonably enter the migration of column catalogue subtree, catalogue subtree is effectively reduced
The number of migration, improve the efficiency for realizing load balancing.
Transmission cost (c) can be used to indicate that unit of transfer's data need the cost spent between node.For c between node
Calculate, can when two meta data servers have data exchange, according to data exchange spend time, calculate this two
Transmission cost between platform meta data server.In view of the stability of network structure, c renewal and infrequently, for node
Between transmission cost can also manually configure and directly specify, for the MDS newly added, by specifying it to some MDS's
Transmission cost, it can be accelerated and incorporate the speed of MDS clusters.
In embodiments of the present invention, the spent time is migrated in order to effectively reduce catalogue subtree, can be according to each section
Transmission cost between point, subarea processing is carried out to each node.By being clustered to c, MDS is divided into different regions,
Transmission cost in region between MDS is significantly less than the transmission cost of the overseas MDS in same district communications, for the ease of the collection of information,
MDS that can be nearest apart from cluster centre using in chosen area is responsible for collecting the information of MDS in local, such as loaded as decision-making MDS
Value and hot value etc., so as to which the information of collection is uploaded into group system in units of region.By group system according to acquisition
Information, the migration of catalogue subtree between each meta data server in region is carried out, to realize the load balancing in region.
Specifically, can be according to all meta data servers in the group system of acquisition each before S103
Corresponding transmission cost value, all meta data servers are subjected to subarea processing；Wherein, include in each region to
A few meta data server.
Accordingly, can be calculated every according to load value corresponding to the meta data server that each region includes
Mean load value corresponding to the individual region；And each institute included according to the mean load value and each region
Load value corresponding to meta data server is stated, selects overload meta data server and all underloadings corresponding to each region
Meta data server.
After the division in each meta data server progress region to group system, using region as processing in subsequent treatment
Unit, to realize the load balancing between each meta data server in the same area.
It should be noted that after drawing regional processing, if load balancing in region can not be realized in a region, can also enter
The trans-regional processing of row, can first call each meta data server in another region with the region arest neighbors, then born
Carry balanced processing.
In multiple metadata that every meta data server is managed, it is possible that first number that access times are more
According to, namely the metadata that hot value is higher, in order to lift the access efficiency of metadata, heat degree threshold can be set, work as metadata
Hot value exceed default heat degree threshold after, then promoter tree copying flow, system creates caching pair on other MDS
This, so as to effectively solve the access problem of hot spot data.
Specifically, all each self-corresponding hot values of metadata in the meta data server can be obtained；Judge whether
The metadata that hot value exceedes preset heat threshold value be present；If so, catalogue subtree corresponding to the metadata is then copied into institute
State in group system and meet in the meta data server of preparatory condition.
Preset heat threshold value can be used to indicate that the higher limit of metadata access number, is somebody's turn to do when the hot value of metadata exceedes
After preset heat threshold value, then illustrate that the access of the metadata is more frequent, can should in order to lift the access efficiency of metadata
Catalogue subtree is copied in other metadata server corresponding to metadata.
Preparatory condition can be the qualifications to meta data server loading condition.When meta data server meets that this is pre-
If during condition, then explanation can create directory the copy of subtree on the meta data server, can also be taken in the metadata
The duplication of catalogue subtree is completed on business device.
In the specific implementation, can according to the sequence number of meta data server itself, successively come choose disclosure satisfy that it is default
The meta data server of condition, when the meta data server for occurring meeting preparatory condition, then created on the meta data server
Build cached copies.
Fig. 3 is a kind of structural representation of meta data management device provided in an embodiment of the present invention, including acquiring unit 31,
Choose unit 32, computing unit 33 and migration units 34；
The acquiring unit 31, for obtaining each self-corresponding load value of all meta data servers in group system and heat
Angle value；
The selection unit 32, for according to load value corresponding to predetermined threshold value and each meta data server,
Select overload meta data server and underloading meta data server；
The computing unit 33, for according to all load values, calculating averagely negative corresponding to the group system
Load value；
The migration units 34, for according to the mean load value, the predetermined threshold value, the overload Metadata Service
Load value and hot value corresponding to load value corresponding to device and hot value and each underloading meta data server, from all
Target metadata server is selected in the underloading meta data server, and respective counts in meta data server are overloaded by described
The catalogue subtree of amount is migrated to the target metadata server.
Optionally, the migration units include the first computation subunit, the second computation subunit, choose subelement and execution
Subelement；
First computation subunit, for according to corresponding to the mean load value and the overload meta data server
Load value, calculate overload quantity corresponding to the overload meta data server；
Second computation subunit, for according to the mean load value, the predetermined threshold value, the overload metadata
Load value and hot value corresponding to load value corresponding to server and hot value and each underloading meta data server, meter
Calculate bearing capacity corresponding to each underloading meta data server；
The selection subelement, for according to bearing capacity corresponding to each underloading meta data server, from all institutes
State and a maximum underloading meta data server of bearing capacity is selected in underloading meta data server as target metadata service
Device；
The execution subelement, for according to bearing capacity corresponding to the overload quantity and the target metadata server,
The catalogue subtree of respective numbers in the overload meta data server is migrated to the target metadata server.
Optionally, second computation subunit is specifically used for according to equation below, calculates all underloading member numbers
According to bearing capacity mL corresponding to jth platform underloading meta data server in server；
Wherein, L represents the predetermined threshold value, and L1 represents the mean load value, PjRepresent jth platform underloading Metadata Service
Hot value corresponding to device, PiRepresent hot value corresponding to the overload meta data server, LjRepresent jth platform underloading metadata clothes
Load value corresponding to business device, LiRepresent load value corresponding to the overload meta data server.
Optionally, in addition to multidomain treat-ment unit；The multidomain treat-ment unit, for according to the group system obtained
In all each self-corresponding transmission cost values of meta data server, all meta data servers are carried out at subregions
Reason；Wherein, at least one meta data server is included in each region；
Accordingly, the computing unit is specifically used for corresponding to the meta data server that includes according to each region
Load value, calculate mean load value corresponding to each region；
The each institute for choosing unit and being specifically used for including according to the mean load value and each region
Load value corresponding to meta data server is stated, selects overload meta data server and all underloadings corresponding to each region
Meta data server.
Optionally, in addition to judging unit and copied cells；
The acquiring unit is additionally operable to obtain each self-corresponding heat of all metadata that the meta data server includes
Angle value；
The judging unit, for judging whether that hot value exceedes the metadata of preset heat threshold value；
If so, the copied cells is then triggered, the copied cells, for catalogue subtree corresponding to the metadata to be answered
Make in the group system and meet in the meta data server of preparatory condition.
The explanation of feature may refer to the related description of embodiment corresponding to Fig. 1 and Fig. 2 in embodiment corresponding to Fig. 3, this
In no longer repeat one by one.
All each self-corresponding load values of meta data server in group system are obtained it can be seen from above-mentioned technical proposal
And hot value；According to load value corresponding to predetermined threshold value and each meta data server, overload meta data server is selected
With underloading meta data server；According to load value corresponding to all meta data servers, can calculate corresponding to group system
One mean load value；According to load value and hot value corresponding to mean load value, predetermined threshold value, overload meta data server with
And load value and hot value corresponding to each underloading meta data server, select one from all underloading meta data servers
The underloading meta data server of catalogue subtree migration is most appropriate for as target metadata server, and will overload metadata clothes
The catalogue subtree of respective numbers is migrated to the target metadata server in business device.In this process, each first number has been taken into full account
According to the temperature situation of server, so as to more reasonably enter the migration of column catalogue subtree, catalogue subtree is effectively reduced
The number of migration, improve the efficiency for realizing load balancing.
Fig. 4 is a kind of hardware architecture diagram of meta data management device 40 provided in an embodiment of the present invention, memory 41,
For storing computer program；Processor 42, for performing the computer program to realize such as above-mentioned metadata management method
The step of.
The embodiment of the present invention additionally provides a kind of computer-readable recording medium, is deposited on the computer-readable recording medium
Computer program is contained, is realized when the computer program is executed by processor such as the step of above-mentioned metadata management method.
A kind of metadata management method, device and the computer-readable recording medium provided above the embodiment of the present invention
It is described in detail.Each embodiment is described by the way of progressive in specification, and what each embodiment stressed is
With the difference of other embodiment, between each embodiment identical similar portion mutually referring to.Disclosed for embodiment
Device for, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is referring to method
Part illustrates.It should be pointed out that for those skilled in the art, before the principle of the invention is not departed from
Put, some improvement and modification can also be carried out to the present invention, these are improved and modification also falls into the guarantor of the claims in the present invention
In the range of shield.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software, the composition and step of each example are generally described according to function in the above description.These
Function is performed with hardware or software mode actually, application-specific and design constraint depending on technical scheme.Specialty
Technical staff can realize described function using distinct methods to each specific application, but this realization should not
Think beyond the scope of this invention.
Directly it can be held with reference to the step of method or algorithm that the embodiments described herein describes with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Claims (10)
- A kind of 1. metadata management method, it is characterised in that including：Obtain each self-corresponding load value of all meta data servers and hot value in group system；According to load value corresponding to predetermined threshold value and each meta data server, select overload meta data server and Underloading meta data server；According to all load values, mean load value corresponding to the group system is calculated；According to load value and hot value corresponding to the mean load value, the predetermined threshold value, the overload meta data server And load value and hot value corresponding to each underloading meta data server, from all underloading meta data servers Target metadata server is selected, and the catalogue subtree of respective numbers in the overload meta data server is migrated to described Target metadata server.
- 2. according to the method for claim 1, it is characterised in that it is described according to the mean load value, the predetermined threshold value, Born corresponding to load value and hot value corresponding to the overload meta data server and each underloading meta data server Load value and hot value, select target metadata server from all underloading meta data servers, and by the overload The catalogue subtree of respective numbers is migrated to the target metadata server and included in meta data server：According to load value corresponding to the mean load value and the overload meta data server, the overload metadata is calculated Overload quantity corresponding to server；According to load value and hot value corresponding to the mean load value, the predetermined threshold value, the overload meta data server And load value and hot value corresponding to each underloading meta data server, calculate each underloading Metadata Service Bearing capacity corresponding to device；According to bearing capacity corresponding to each underloading meta data server, chosen from all underloading meta data servers Go out a maximum underloading meta data server of bearing capacity as target metadata server；According to bearing capacity corresponding to the overload quantity and the target metadata server, by the overload meta data server The catalogue subtree of respective numbers is migrated to the target metadata server.
- 3. according to the method for claim 2, it is characterised in that it is described according to the mean load value, the predetermined threshold value, Born corresponding to load value and hot value corresponding to the overload meta data server and each underloading meta data server Load value and hot value, calculate each underloading meta data server corresponding to bearing capacity include：According to equation below, calculate in all underloading meta data servers corresponding to jth platform underloading meta data server Bearing capacity mL；<mrow> <mi>m</mi> <mi>L</mi> <mo>=</mo> <mrow> <mo>(</mo> <mi>L</mi> <mo>-</mo> <mi>L</mi> <mn>1</mn> <mo>)</mo> </mrow> <mo>*</mo> <mrow> <mo>(</mo> <mfrac> <mrow> <msub> <mi>P</mi> <mi>j</mi> </msub> <mo>*</mo> <msub> <mi>L</mi> <mi>i</mi> </msub> </mrow> <mrow> <msub> <mi>P</mi> <mi>i</mi> </msub> <mo>*</mo> <msub> <mi>L</mi> <mi>j</mi> </msub> </mrow> </mfrac> <mo>)</mo> </mrow> <mo>;</mo> </mrow>Wherein, L represents the predetermined threshold value, and L1 represents the mean load value, PjRepresent jth platform underloading meta data server pair The hot value answered, PiRepresent hot value corresponding to the overload meta data server, LjRepresent jth platform underloading meta data server Corresponding load value, LiRepresent load value corresponding to the overload meta data server.
- 4. according to the method for claim 1, it is characterised in that described according to all load values, calculate described Also include before mean load value corresponding to group system：According to all each self-corresponding transmission cost values of meta data server in the group system obtained, by all institutes State meta data server and carry out subarea processing；Wherein, at least one meta data server is included in each region；Accordingly, it is described according to all load values, calculate the group system corresponding to mean load value include：Load value corresponding to the meta data server included according to each region, calculate corresponding to each region Mean load value；It is described according to load value corresponding to the mean load value and each meta data server, select the first number of overload Include according to server and all underloading meta data servers：Load corresponding to each meta data server included according to the mean load value and each region Value, select overload meta data server and all underloading meta data servers corresponding to each region.
- 5. according to the method described in claim 1-4 any one, it is characterised in that also include：Obtain all each self-corresponding hot values of metadata in the meta data server；Judge whether that hot value exceedes the metadata of preset heat threshold value；If so, catalogue subtree corresponding to the metadata is then copied in the group system to the metadata for meeting preparatory condition In server.
- 6. a kind of meta data management device, it is characterised in that including acquiring unit, choose unit, computing unit and migration units；The acquiring unit, for obtaining each self-corresponding load value of all meta data servers in group system and hot value；The selection unit, for according to load value corresponding to predetermined threshold value and each meta data server, selecting Overload meta data server and underloading meta data server；The computing unit, for according to all load values, calculating mean load value corresponding to the group system；The migration units, for corresponding according to the mean load value, the predetermined threshold value, the overload meta data server Load value and hot value and each underloading meta data server corresponding to load value and hot value, from all described light Carry in meta data server and select target metadata server, and by the mesh of respective numbers in the overload meta data server Record subtree is migrated to the target metadata server.
- 7. device according to claim 6, it is characterised in that also including multidomain treat-ment unit；The multidomain treat-ment unit, For all each self-corresponding transmission cost values of meta data server in the group system according to acquisition, by all institutes State meta data server and carry out subarea processing；Wherein, at least one meta data server is included in each region；Accordingly, loaded corresponding to the meta data server that the computing unit is specifically used for including according to each region Value, calculates mean load value corresponding to each region；The each member chosen unit and be specifically used for including according to the mean load value and each region Load value corresponding to data server, select overload meta data server and all underloading member numbers corresponding to each region According to server.
- 8. the device according to claim 6 or 7, it is characterised in that also including judging unit and copied cells；The acquiring unit is additionally operable to obtain each self-corresponding hot value of all metadata that the meta data server includes；The judging unit, for judging whether that hot value exceedes the metadata of preset heat threshold value；If so, the copied cells is then triggered, the copied cells, for catalogue subtree corresponding to the metadata to be copied to Meet in the group system in the meta data server of preparatory condition.
- A kind of 9. meta data management device, it is characterised in that including：Memory, for storing computer program；Processor, for performing the computer program to realize the metadata management side as described in claim 1 to 5 any one The step of method.
- 10. a kind of computer-readable recording medium, it is characterised in that be stored with computer on the computer-readable recording medium Program, realizing the metadata management method as described in any one of claim 1 to 5 when the computer program is executed by processor Step.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711116398.5A CN107766159A (en) | 2017-11-13 | 2017-11-13 | A kind of metadata management method, device and computer-readable recording medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711116398.5A CN107766159A (en) | 2017-11-13 | 2017-11-13 | A kind of metadata management method, device and computer-readable recording medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766159A true CN107766159A (en) | 2018-03-06 |
Family
ID=61272761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711116398.5A Pending CN107766159A (en) | 2017-11-13 | 2017-11-13 | A kind of metadata management method, device and computer-readable recording medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766159A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN109634915A (en) * | 2018-11-28 | 2019-04-16 | 深圳市网心科技有限公司 | File dispositions method, Cloud Server, system and storage medium |
| CN111190863A (en) * | 2019-12-29 | 2020-05-22 | 浪潮电子信息产业股份有限公司 | Catalog management method, device, equipment and medium |
| CN111459407A (en) * | 2020-03-12 | 2020-07-28 | 苏州浪潮智能科技有限公司 | Data equalization method and system based on MDS subtree equalization algorithm |
| CN111580963A (en) * | 2020-04-29 | 2020-08-25 | 平安科技（深圳）有限公司 | Load balancing method and device for storage cluster, computer equipment and storage medium |
| CN112256438A (en) * | 2020-06-28 | 2021-01-22 | 腾讯科技（深圳）有限公司 | Load balancing control method and device, storage medium and electronic equipment |
| CN112988062A (en) * | 2021-01-28 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN113055448A (en) * | 2021-02-28 | 2021-06-29 | 新华三信息技术有限公司 | Metadata management method and device |
| CN113360455A (en) * | 2021-07-16 | 2021-09-07 | 北京天融信网络安全技术有限公司 | Data processing method, device, equipment and medium of super-fusion system |
| CN113407108A (en) * | 2020-03-16 | 2021-09-17 | 北京沃东天骏信息技术有限公司 | Data storage method and system |
| CN113608876A (en) * | 2021-08-12 | 2021-11-05 | 中国科学技术大学 | Distributed file system metadata load balancing method based on load type perception |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| US7861305B2 (en) * | 2007-02-07 | 2010-12-28 | International Business Machines Corporation | Method and system for hardware based program flow monitor for embedded software |
| CN103279568A (en) * | 2013-06-18 | 2013-09-04 | 无锡紫光存储系统有限公司 | System and method for metadata management |
| CN103294785A (en) * | 2013-05-17 | 2013-09-11 | 华中科技大学 | Packet-based metadata server cluster management method |
| CN103763365A (en) * | 2014-01-16 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method and system for load balancing of metadata service under cloud storage |
| CN107145307A (en) * | 2017-04-27 | 2017-09-08 | 郑州云海信息技术有限公司 | A kind of dynamic metadata optimization method and system based on distributed storage |
-
2017
- 2017-11-13 CN CN201711116398.5A patent/CN107766159A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7861305B2 (en) * | 2007-02-07 | 2010-12-28 | International Business Machines Corporation | Method and system for hardware based program flow monitor for embedded software |
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| CN103294785A (en) * | 2013-05-17 | 2013-09-11 | 华中科技大学 | Packet-based metadata server cluster management method |
| CN103279568A (en) * | 2013-06-18 | 2013-09-04 | 无锡紫光存储系统有限公司 | System and method for metadata management |
| CN103763365A (en) * | 2014-01-16 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method and system for load balancing of metadata service under cloud storage |
| CN107145307A (en) * | 2017-04-27 | 2017-09-08 | 郑州云海信息技术有限公司 | A kind of dynamic metadata optimization method and system based on distributed storage |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN108509275B (en) * | 2018-03-29 | 2019-11-12 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN109634915A (en) * | 2018-11-28 | 2019-04-16 | 深圳市网心科技有限公司 | File dispositions method, Cloud Server, system and storage medium |
| CN111190863A (en) * | 2019-12-29 | 2020-05-22 | 浪潮电子信息产业股份有限公司 | Catalog management method, device, equipment and medium |
| CN111190863B (en) * | 2019-12-29 | 2022-04-22 | 浪潮电子信息产业股份有限公司 | Catalog management method, device, equipment and medium |
| CN111459407A (en) * | 2020-03-12 | 2020-07-28 | 苏州浪潮智能科技有限公司 | Data equalization method and system based on MDS subtree equalization algorithm |
| CN111459407B (en) * | 2020-03-12 | 2023-05-16 | 苏州浪潮智能科技有限公司 | Data equalization method and system based on MDS sub-tree equalization algorithm |
| CN113407108A (en) * | 2020-03-16 | 2021-09-17 | 北京沃东天骏信息技术有限公司 | Data storage method and system |
| CN111580963B (en) * | 2020-04-29 | 2023-07-11 | 平安科技（深圳）有限公司 | Load balancing method and device for storage cluster, computer equipment and storage medium |
| WO2021217864A1 (en) * | 2020-04-29 | 2021-11-04 | 平安科技（深圳）有限公司 | Load balancing method and apparatus for storage cluster, computer device, and storage medium |
| CN111580963A (en) * | 2020-04-29 | 2020-08-25 | 平安科技（深圳）有限公司 | Load balancing method and device for storage cluster, computer equipment and storage medium |
| CN112256438A (en) * | 2020-06-28 | 2021-01-22 | 腾讯科技（深圳）有限公司 | Load balancing control method and device, storage medium and electronic equipment |
| CN112988062B (en) * | 2021-01-28 | 2023-02-14 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN112988062A (en) * | 2021-01-28 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN113055448A (en) * | 2021-02-28 | 2021-06-29 | 新华三信息技术有限公司 | Metadata management method and device |
| CN113360455A (en) * | 2021-07-16 | 2021-09-07 | 北京天融信网络安全技术有限公司 | Data processing method, device, equipment and medium of super-fusion system |
| CN113360455B (en) * | 2021-07-16 | 2024-02-02 | 北京天融信网络安全技术有限公司 | Data processing method, device, equipment and medium of super fusion system |
| CN113608876A (en) * | 2021-08-12 | 2021-11-05 | 中国科学技术大学 | Distributed file system metadata load balancing method based on load type perception |
| CN113608876B (en) * | 2021-08-12 | 2024-03-29 | 中国科学技术大学 | Distributed file system metadata load balancing method based on load type perception |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766159A (en) | A kind of metadata management method, device and computer-readable recording medium | |
| CN103139302B (en) | Real-time copy scheduling method considering load balancing | |
| CN101370030B (en) | Resource load stabilization method based on contents duplication | |
| CN108509275B (en) | A kind of catalogue moving method and metadata load-balancing method | |
| CN105242983B (en) | A kind of date storage method and a kind of data storage management service device | |
| CN103929454B (en) | The method and system of load balancing storage in a kind of cloud computing platform | |
| CN112256438B (en) | Load balancing control method and device, storage medium and electronic equipment | |
| CN103178989B (en) | Access hot statistics method and device | |
| CN108762924A (en) | A kind of method, apparatus and computer readable storage medium of load balancing | |
| CN104539730B (en) | Towards the load-balancing method of video in a kind of HDFS | |
| CN107729514A (en) | A kind of Replica placement node based on hadoop determines method and device | |
| CN103077197A (en) | Data storing method and device | |
| WO2017016499A1 (en) | Method and device for leveling load of distributed database | |
| CN108900626A (en) | Date storage method, apparatus and system under a kind of cloud environment | |
| CN107623732A (en) | A kind of date storage method based on cloud platform, device, equipment and storage medium | |
| CN107609140A (en) | A kind of method and device of distributive catalogue of document system file access | |
| CN101179494A (en) | Resource distribution method facing to network multimedia transmission service | |
| Rajalakshmi et al. | An improved dynamic data replica selection and placement in cloud | |
| CN108376103A (en) | A kind of the equilibrium of stock control method and server of cloud platform | |
| JP2019121334A (en) | Data storage and dynamic migration method, and data storage and dynamic migration device | |
| CN104158902B (en) | A kind of Hbase data blocks distribution method and device based on number of request | |
| CN107395708A (en) | A kind of method and apparatus for handling download request | |
| CN107453948A (en) | The storage method and system of a kind of network measurement data | |
| CN110990366A (en) | Index allocation method and device for improving performance of log system based on ES | |
| CN110321225A (en) | Load-balancing method, meta data server and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180306 |
|
| RJ01 | Rejection of invention patent application after publication |