CN105389392A - Metadata load statistical method and system - Google Patents
Metadata load statistical method and system Download PDFInfo
- Publication number
- CN105389392A CN105389392A CN201510961006.XA CN201510961006A CN105389392A CN 105389392 A CN105389392 A CN 105389392A CN 201510961006 A CN201510961006 A CN 201510961006A CN 105389392 A CN105389392 A CN 105389392A
- Authority
- CN
- China
- Prior art keywords
- metadata
- cpu
- value
- occupation value
- current mds
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/503—Resource availability
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a metadata load statistical method and system, applied to a distributed file system of a user state. The metadata load statistical method comprises the steps of obtaining an amount of metadata requests of a current MDS process; obtaining a CPU core where the current MDS process locates; calculating a CPU occupation value; obtaining a memory usage value of the current MDS process; and counting a metadata load value of the current MDS process based on the metadata requesting amount, the CPU occupation value and the memory usage value. According to the metadata load statistical method, the load value of the metadata process is obtained by obtaining three key factors of the metadata load, so that statistics of metadata load can be performed exactly and efficiently.
Description
Technical field
The present invention relates to distributed file system field, particularly relate to a kind of metadata load statistics method and system.
Background technology
At present, distributed file system has obtained production application widely, in distributed file system, Metadata Service maintains the directory space of whole file system, metadata is the information of data of description attribute, and small documents application can produce a large amount of metadata operations, generally promote metadata of distributed type file system performance by metadata cluster.
But the difficult problem that metadata cluster faces is exactly how correctly to add up metadata load effectively to carry out load balance.
Summary of the invention
In view of this, fundamental purpose of the present invention is to provide a kind of metadata load statistics method and system, can the load of precise and high efficiency geo-statistic metadata.
For achieving the above object, the invention provides a kind of metadata load statistics method, be applied to the distributed file system of User space, comprise:
Obtain the metadata request quantity of current mds process;
Obtain the CPU core at described current mds process place, calculate CPU occupation value;
Obtain the EMS memory occupation value of described current mds process;
The metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
Preferably, also comprise before obtaining the metadata request quantity of current mds process:
For the CPU core of operating system corresponding to bind meta-data.
Preferably, described metadata request comprises getattr, setattr, getxattr, setxattr, stat and mkdir.
Preferably, calculate CPU occupation value to comprise:
Obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
Preferably, according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics, the metadata load value of current mds process comprises:
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value.
Present invention also offers a kind of metadata load statistics system, be applied to the distributed file system of User space, comprise:
Metadata request quantity determination module, for obtaining the metadata request quantity of current mds process;
CPU occupation value determination module, for obtaining the CPU core at described current mds process place, calculates CPU occupation value;
EMS memory occupation value determination module, for obtaining the EMS memory occupation value of described current mds process;
Metadata load value statistical module, for the metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
Preferably, described metadata load statistics system also comprises:
CPU core binding module, for the CPU core for operating system corresponding to bind meta-data.
Preferably, described metadata request comprises getattr, setattr, getxattr, setxattr, stat and mkdir.
Preferably, described CPU occupation value determination module specifically for:
Obtain the CPU core at described current mds process place, obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
Preferably, described metadata load value statistical module specifically for:
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value.
Apply a kind of metadata load statistics method and system provided by the invention, be applied to the distributed file system of User space, obtain the metadata request quantity of current mds process, obtain the CPU core at described current mds process place, calculate CPU occupation value, obtain the EMS memory occupation value of described current mds process, according to described metadata request quantity, the metadata load value of current mds process described in described CPU occupation value and described EMS memory occupation Data-Statistics, this method obtains the load value of metadata process by 3 key factors obtaining metadata load, the statistics of metadata load can be carried out accurately and efficiently.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
Fig. 1 is the process flow diagram of a kind of metadata load statistics of the present invention embodiment of the method one;
Fig. 2 is the structural representation of a kind of metadata load statistics of the present invention system embodiment two.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out
Describe clearly and completely, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Embodiment one:
The invention provides a kind of metadata load statistics method, be applied to the distributed file system of User space, Fig. 1 shows the process flow diagram of metadata load statistics embodiment of the method one of the present invention, comprising:
Step S101: the metadata request quantity obtaining current mds process;
Obtain distributed file system metadata service device end mds process current time metadata request OP quantity, MDS request is all the operation of regarding metadata, comprises the operations such as getattr, setattr, getxattr, setxattr, stat and mkdir, wherein:
Getattr statement is used for returning an Integer, and this is the attribute of file, catalogue or a file;
SetAttr statement is used for being that a file sets a property information;
Getxattr statement is for listing the value corresponding to extended attribute;
Setxattr statement is used for arranging extended attribute;
Sat statement is for returning a message structure relevant with object naming file;
Mkdir statement is for creating the catalogue of the title of specifying.
Step S102: the CPU core obtaining described current mds process place, calculates CPU occupation value;
A CPU core of direct bindings system in process metadata process, save and find the time which CPU core mds process takies, after determining CPU core, obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
Step S103: the EMS memory occupation value obtaining described current mds process;
Obtain current mds process committed memory size MEM.
Step S104: the metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value, if MDS
loadfor current mds process load value, computing formula is:
MDS
load＝OP+CPU*100+MEM。
A kind of metadata load statistics method that application the present embodiment provides, be applied to the distributed file system of User space, obtain the metadata request quantity of current mds process, obtain the CPU core at described current mds process place, calculate CPU occupation value, obtain the EMS memory occupation value of described current mds process, according to described metadata request quantity, the metadata load value of current mds process described in described CPU occupation value and described EMS memory occupation Data-Statistics, this method obtains the load value of metadata process by 3 key factors obtaining metadata load, the statistics of metadata load can be carried out accurately and efficiently.
Embodiment two:
Present invention also offers a kind of metadata load statistics system, be applied to the distributed file system of User space, Fig. 2 shows the structural representation of metadata load statistics system embodiment two of the present invention, comprising:
Metadata request quantity determination module 101, for obtaining the metadata request quantity of current mds process;
CPU occupation value determination module 102, for obtaining the CPU core at described current mds process place, calculates CPU occupation value;
EMS memory occupation value determination module 103, for obtaining the EMS memory occupation value of described current mds process;
Metadata load value statistical module 104, for the metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
The present embodiment also can comprise:
CPU core binding module, for the CPU core for operating system corresponding to bind meta-data.
Described in the present embodiment, metadata request comprises getattr, setattr, getxattr, setxattr, stat and mkdir.
Described CPU occupation value determination module 102 specifically for:
Obtain the CPU core at described current mds process place, obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
Described metadata load value statistical module 104 specifically for:
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value.
A kind of metadata load statistics system that application the present embodiment provides, be applied to the distributed file system of User space, obtain the metadata request quantity of current mds process, obtain the CPU core at described current mds process place, calculate CPU occupation value, obtain the EMS memory occupation value of described current mds process, according to described metadata request quantity, the metadata load value of current mds process described in described CPU occupation value and described EMS memory occupation Data-Statistics, this method obtains the load value of metadata process by 3 key factors obtaining metadata load, the statistics of metadata load can be carried out accurately and efficiently.
It should be noted that, each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually see.For system class embodiment, due to itself and embodiment of the method basic simlarity, so description is fairly simple, relevant part illustrates see the part of embodiment of the method.
Finally, also it should be noted that, in this article, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.
Be described in detail method and system provided by the present invention above, apply specific case herein and set forth principle of the present invention and embodiment, the explanation of above embodiment just understands method of the present invention and core concept thereof for helping; Meanwhile, for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (10)
1. a metadata load statistics method, is applied to the distributed file system of User space, it is characterized in that, comprising:
Obtain the metadata request quantity of current mds process;
Obtain the CPU core at described current mds process place, calculate CPU occupation value;
Obtain the EMS memory occupation value of described current mds process;
The metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
2. metadata load statistics method according to claim 1, is characterized in that, also comprises before obtaining the metadata request quantity of current mds process:
For the CPU core of operating system corresponding to bind meta-data.
3. metadata load statistics method according to claim 1, it is characterized in that, described metadata request comprises getattr, setattr, getxattr, setxattr, stat and mkdir.
4. metadata load statistics method according to claim 1, is characterized in that, calculates CPU occupation value and comprises:
Obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
5. metadata load statistics method according to claim 1, is characterized in that, according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics, the metadata load value of current mds process comprises:
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value.
6. a metadata load statistics system, is applied to the distributed file system of User space, it is characterized in that, comprising:
Metadata request quantity determination module, for obtaining the metadata request quantity of current mds process;
CPU occupation value determination module, for obtaining the CPU core at described current mds process place, calculates CPU occupation value;
EMS memory occupation value determination module, for obtaining the EMS memory occupation value of described current mds process;
Metadata load value statistical module, for the metadata load value of current mds process according to described metadata request quantity, described CPU occupation value and described EMS memory occupation Data-Statistics.
7. metadata load statistics system according to claim 6, is characterized in that, also comprise:
CPU core binding module, for the CPU core for operating system corresponding to bind meta-data.
8. metadata load statistics system according to claim 6, it is characterized in that, described metadata request comprises getattr, setattr, getxattr, setxattr, stat and mkdir.
9. metadata load statistics system according to claim 6, is characterized in that, described CPU occupation value determination module specifically for:
Obtain the CPU core at described current mds process place, obtain the CPU usage that described current mds process takies described CPU core, described CPU usage is multiplied by 100 and obtains CPU occupation value.
10. metadata load statistics system according to claim 6, is characterized in that, described metadata load value statistical module specifically for:
Described request of data quantity, described CPU occupation value and described EMS memory occupation value are carried out addition and obtains metadata load value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510961006.XA CN105389392A (en) | 2015-12-18 | 2015-12-18 | Metadata load statistical method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510961006.XA CN105389392A (en) | 2015-12-18 | 2015-12-18 | Metadata load statistical method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105389392A true CN105389392A (en) | 2016-03-09 |
Family
ID=55421682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510961006.XA Pending CN105389392A (en) | 2015-12-18 | 2015-12-18 | Metadata load statistical method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105389392A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055448A (en) * | 2021-02-28 | 2021-06-29 | 新华三信息技术有限公司 | Metadata management method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| CN102223419A (en) * | 2011-07-05 | 2011-10-19 | 北京邮电大学 | Virtual resource dynamic feedback balanced allocation mechanism for network operation system |
| US20120150930A1 (en) * | 2010-12-10 | 2012-06-14 | Electronics And Telecommunications Research Institute | Cloud storage and method for managing the same |
| CN102523158A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Metadata server cluster load balancing method based on weight |
| CN102904949A (en) * | 2012-10-08 | 2013-01-30 | 华中科技大学 | Replica-based dynamic metadata cluster system |
| CN104168332A (en) * | 2014-09-01 | 2014-11-26 | 广东电网公司信息中心 | Load balance and node state monitoring method in high performance computing |
| CN105094992A (en) * | 2015-09-25 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | File request processing method and system |
-
2015
- 2015-12-18 CN CN201510961006.XA patent/CN105389392A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| US20120150930A1 (en) * | 2010-12-10 | 2012-06-14 | Electronics And Telecommunications Research Institute | Cloud storage and method for managing the same |
| CN102223419A (en) * | 2011-07-05 | 2011-10-19 | 北京邮电大学 | Virtual resource dynamic feedback balanced allocation mechanism for network operation system |
| CN102523158A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Metadata server cluster load balancing method based on weight |
| CN102904949A (en) * | 2012-10-08 | 2013-01-30 | 华中科技大学 | Replica-based dynamic metadata cluster system |
| CN104168332A (en) * | 2014-09-01 | 2014-11-26 | 广东电网公司信息中心 | Load balance and node state monitoring method in high performance computing |
| CN105094992A (en) * | 2015-09-25 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | File request processing method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055448A (en) * | 2021-02-28 | 2021-06-29 | 新华三信息技术有限公司 | Metadata management method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105786857A (en) | Method and system for improving video aggregation efficiency | |
| CN105389392A (en) | Metadata load statistical method and system | |
| CN105095102A (en) | Automatic expansion method and device for snapshot volume | |
| CN106161498A (en) | The multi-platform dissemination method of a kind of contribution and system | |
| CN202524426U (en) | Enterprise cloud computing service application system | |
| CN105095478B (en) | The method and device of the creation time of file in a kind of acquisition distributed file system | |
| Brunozzi | Big data and nosql with amazon dynamodb | |
| CN105677619A (en) | Method and device for adjusting paragraph spacing | |
| CN104581818A (en) | Traffic exchange method and system based on mobile terminal | |
| Eldjalil et al. | A novel pricing policy for G2V and V2G services | |
| CN206054492U (en) | Handle nut structure | |
| CN203644072U (en) | Power transmission line information display system | |
| CN103716256A (en) | Method and device for selecting basic devices for network content services | |
| CN201716796U (en) | Distributed calculating device based on fiscal cash register | |
| CN114328558A (en) | List updating method, apparatus, device and storage medium | |
| CN113760900A (en) | Method and device for real-time data summarization and interval summarization | |
| Noble | C++ code referred to in Funk, Mayhew, Noble," How many delta-matroids are there?" | |
| Xu | Green internet datacenters with dynamic and diverse traffic | |
| CN104423910A (en) | Cross-domain page display method, device and terminal equipment | |
| Afonso | An Elasticity Controller for Applications Orquestrated with Cloudify/Um Controlador de Elasticidade Para Aplicações Orquestradas Com Cloudify | |
| Chao | Spanner Database Trial | |
| Xu | A Substation Expansion of Design and Implementation | |
| Lomotey et al. | Energy Consumption Analysis in Ubiquitous Cloud Computing | |
| Chang et al. | Demonstration of lmmp automated performance testing using cloud computing architecture | |
| CN103259854A (en) | Quantum streaming media cloud server based on embedded system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160309 |