CN102567089A - Design method for thread pool of metadata server in distributed file system - Google Patents
Design method for thread pool of metadata server in distributed file system Download PDFInfo
- Publication number
- CN102567089A CN102567089A CN2011103260539A CN201110326053A CN102567089A CN 102567089 A CN102567089 A CN 102567089A CN 2011103260539 A CN2011103260539 A CN 2011103260539A CN 201110326053 A CN201110326053 A CN 201110326053A CN 102567089 A CN102567089 A CN 102567089A
- Authority
- CN
- China
- Prior art keywords
- thread
- affairs
- file system
- distributed file
- stage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a thread pool mechanism of a metadata server in a distributed file system, which aims to improve processing capacity of the metadata server in the distributed file system, increase converged input/output operations per second of the metadata server in the distributed file system and satisfy frequent metadata server requests in the distributed file system. By the aid of the mechanism, decrease of the processing capacity of the metadata server caused by transaction blockage is avoided by using various methods, and a negative condition like thread wait blockage, unbalanced load or the like possibly caused by repeated transaction submittal is considered.
Description
Technical field
The present invention relates to the thread pool mechanism in the distributed file system, specifically, relate to the method for designing of meta data server thread pool in a kind of distributed file system.
Background technology
In distributed file system, the task of meta data server is the services request that other nodes provide accesses meta-data in system.In the typical application of most of distributed file system, like the internet, Streaming Media etc. are to very frequent of the access request of meta data server.So the speed of meta data server response and processing services request directly influences the performance of whole file system in the distributed file system.
In distributed file system; Because the access request to metadata has polytype; The entire process process of a lot of requests often needs the cooperation of a plurality of parts to accomplish; Picture daily record, data server etc., also there is the processing speed of some other behavioral implications server thread the while, as the slow disk operating of access speed, network service and lock competition.In traditional block type task transaction module, serve sometimes after thread must wait for that some tasks is accomplished, can carry out next step operation.Usually, in metadata system, the quantity of service thread is limited, if thereby service the thread most of the time be in the state of getting clogged, then can cause other request to have no chance to be processed; If become big to number of threads, then can bring the thread overhead in switching on the one hand, on the other hand, number of threads is too much, can bring the situation of more serious lock competition.Thus, for a distributed file system of hoping the request of processing magnanimity, design a kind of suitable thread pool model and avoid this problem, and then improve the response process speed of meta data server effectively, be necessary.
Summary of the invention
The present invention introduces a kind of thread pool mechanism, and the purpose of this mechanism is to improve the speed of meta data server response and process metadata request, satisfies metadata services request frequent in the distributed file system.
The method for designing of meta data server thread pool in a kind of distributed file system; Request each time in the meta data server is as affairs; Describe its state by the state description module, each things is carried out stage by stage, and the execution in each stage is without any need for obstruction.
Preferably, the thread in the said thread pool is the unblock model, and thread can not waited for disk operating or network service.
Preferably, after a stage of said affairs was complete, thread need be confirmed next step behavior according to its rreturn value.
Preferably; If the stage of said affairs is in the process of implementation because IO gets clogged; Then transfer to the asynchronous execution of asynchronous thread; After IO satisfied condition again, the IO execution thread was delivered to these affairs on the transaction queues of service processing thread again, when the formation of thread cyclic access things, was carried out again.
Preferably, said thread all is set to active state, and the active threads in the said thread pool is dynamically adjusted according to task amount.
Preferably, the thread in the said thread pool all has the transaction queues of oneself, the load of this thread of length mark through transaction queues.
Preferably, when the transaction queues of thread is submitted affairs to, the thread in the thread pool is adopted polling mode.
The present invention introduces meta data server thread pool mechanism in a kind of distributed file system; The purpose of this mechanism is to improve the processing power of metadata of distributed type file system server; Increase its polymerization IOPS, satisfy metadata services request frequent in the distributed file system.Negative condition such as this mechanism descends the meta data server processing power of avoiding causing because of affairs obstruction etc. that ins all sorts of ways, and thread waits lock of also having considered simultaneously in affairs are repeatedly submitted to, possibly to cause or load are unbalanced.
Embodiment
The present invention introduces a kind of thread pool mechanism, and the purpose of this mechanism is to improve the speed of meta data server response and process metadata request, satisfies metadata services request frequent in the distributed file system.
Multithreading can improve the efficient of issued transaction; But when meta data server provides service; Tend to the wait that gets clogged because waiting for disk I or network I/O, and adopt the mode that increases number of threads, the thread switching that it is frequent etc. can influence the service performance of system; Based on this, the present invention proposes following thread pool mechanism and be used for the time waste of as far as possible avoiding unnecessary:
(1) notion of introducing affairs in the whole meta data server, ask has special structure to explain its state as affairs at every turn.For each affairs, be some stages with its cutting, the execution in each stage is all without any need for obstruction.
(2) thread in the service thread pool is the unblock model, and thread can not waited for disk operating consuming time or network service.After certain stage of affairs accomplishes; The service thread need be confirmed next step behavior based on its return of value: if return of value was (after EWOULDBLOCK) expression should be accomplished in the stage; After needing to wait for that disk operating or network operation are accomplished; Could get into next transaction phase, then the current service thread is abandoned continuing to handle these affairs, then goes to carry out other affairs; If return of value is non--EWOULDBLOCK, represent that then the current service thread can continue to handle this affairs, so continue to handle the next stage of these affairs.
(3) transaction phase that gets clogged is by the asynchronous execution of other threads; By the time IO accomplishes again and satisfies; This IO execution thread can be delivered these affairs on the transaction queues to the service processing thread again; In thread cyclic access transaction queues, the affairs of blocking in the past can be continued to carry out by thread again like this.Adopt this mode, can avoid the service processing thread to get clogged, make the service thread can be always process metadata association requests ceaselessly, effectively improve the processing power of service thread.
(4) for each service thread active state is set; The number of active threads comes dynamically adjustment according to task amount in the thread pool; When metadata request was fewer, the number of threads of participating in service was quite less, promptly when task is delivered; Selection is in movable thread and accomplishes issued transaction, can reduce the time cost that system thread switches like this; When the metadata service role became big, the thread that need the part in the thread pool be in dormant state activated, and handles.
(5) each thread in the thread pool has the transaction queues of oneself, indicates the load of this thread through the length of transaction queues.When the transaction queues of thread is submitted affairs to, the thread in the thread pool is adopted the mode of poll, thread occurs and frequently switch because of competing same formation to avoid occurring, shine violent decline into performance.Simultaneously, so also can guarantee load balancing, avoid occurring a large amount of affairs and be delivered on the same thread, cause much busier not busy situation and influence the processing power of meta data server at system's each processing threads of run duration.
Claims (7)
1. the method for designing of meta data server thread pool in the distributed file system; It is characterized in that: the request each time in the meta data server is as affairs; Describe its state by the state description module, each things is carried out stage by stage, and the execution in each stage is without any need for obstruction.
2. the method for claim 1, it is characterized in that: the thread in the said thread pool is the unblock model, and thread can not waited for disk operating or network service.
3. the method for claim 1, it is characterized in that: after a stage of said affairs was complete, thread need be confirmed next step behavior according to its rreturn value.
4. the method for claim 1; It is characterized in that: if the stage of said affairs is in the process of implementation because IO gets clogged; Then transfer to the asynchronous execution of asynchronous thread; After IO satisfied condition again, the IO execution thread was delivered to these affairs on the transaction queues of service processing thread again, when the formation of thread cyclic access things, was carried out again.
5. the method for claim 1, it is characterized in that: said thread all is set to active state, and the active threads in the said thread pool is dynamically adjusted according to task amount.
6. the method for claim 1, it is characterized in that: the thread in the said thread pool all has the transaction queues of oneself, the load of this thread of length mark through transaction queues.
7. method as claimed in claim 6 is characterized in that: when the transaction queues of thread is submitted affairs to, the thread in the thread pool is adopted polling mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110326053.9A CN102567089B (en) | 2011-10-25 | 2011-10-25 | Design method for thread pool of metadata server in distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110326053.9A CN102567089B (en) | 2011-10-25 | 2011-10-25 | Design method for thread pool of metadata server in distributed file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102567089A true CN102567089A (en) | 2012-07-11 |
| CN102567089B CN102567089B (en) | 2014-02-19 |
Family
ID=46412585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110326053.9A Active CN102567089B (en) | 2011-10-25 | 2011-10-25 | Design method for thread pool of metadata server in distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102567089B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103327128A (en) * | 2013-07-23 | 2013-09-25 | 百度在线网络技术（北京）有限公司 | Intermediate data transmission method and system for MapReduce |
| CN103457997A (en) * | 2013-07-01 | 2013-12-18 | 曙光信息产业(北京)有限公司 | Data transmission method and system |
| CN104850460A (en) * | 2015-06-02 | 2015-08-19 | 上海斐讯数据通信技术有限公司 | Service program thread management method |
| CN106469018A (en) * | 2015-08-17 | 2017-03-01 | 阿里巴巴集团控股有限公司 | The load monitoring method and apparatus of distributed memory system |
| CN106648832A (en) * | 2016-12-09 | 2017-05-10 | 武汉烽火信息集成技术有限公司 | Virtual machine resource utilization improving device and method thereof |
| CN107346265A (en) * | 2016-05-05 | 2017-11-14 | 北京忆恒创源科技有限公司 | Realize QoS method and apparatus |
| CN107528924A (en) * | 2017-10-09 | 2017-12-29 | 郑州云海信息技术有限公司 | A kind of distributed type assemblies Metadata Service dispositions method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1834956A (en) * | 2005-03-18 | 2006-09-20 | 联想(北京)有限公司 | Processing of multiroute processing element data |
| CN101464810A (en) * | 2008-12-31 | 2009-06-24 | 金蝶软件（中国）有限公司 | Service program processing method and server |
| CN101908000A (en) * | 2010-07-08 | 2010-12-08 | 北京航空航天大学 | Method for blocking and awakening transaction threads in hardware transactional memory system |
| CN101997854A (en) * | 2009-08-31 | 2011-03-30 | 阿里巴巴集团控股有限公司 | Processing system and method for providing data service |
-
2011
- 2011-10-25 CN CN201110326053.9A patent/CN102567089B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1834956A (en) * | 2005-03-18 | 2006-09-20 | 联想(北京)有限公司 | Processing of multiroute processing element data |
| CN101464810A (en) * | 2008-12-31 | 2009-06-24 | 金蝶软件（中国）有限公司 | Service program processing method and server |
| CN101997854A (en) * | 2009-08-31 | 2011-03-30 | 阿里巴巴集团控股有限公司 | Processing system and method for providing data service |
| CN101908000A (en) * | 2010-07-08 | 2010-12-08 | 北京航空航天大学 | Method for blocking and awakening transaction threads in hardware transactional memory system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103457997A (en) * | 2013-07-01 | 2013-12-18 | 曙光信息产业(北京)有限公司 | Data transmission method and system |
| CN103327128A (en) * | 2013-07-23 | 2013-09-25 | 百度在线网络技术（北京）有限公司 | Intermediate data transmission method and system for MapReduce |
| CN104850460A (en) * | 2015-06-02 | 2015-08-19 | 上海斐讯数据通信技术有限公司 | Service program thread management method |
| CN106469018A (en) * | 2015-08-17 | 2017-03-01 | 阿里巴巴集团控股有限公司 | The load monitoring method and apparatus of distributed memory system |
| CN106469018B (en) * | 2015-08-17 | 2019-12-27 | 阿里巴巴集团控股有限公司 | Load monitoring method and device for distributed storage system |
| CN107346265A (en) * | 2016-05-05 | 2017-11-14 | 北京忆恒创源科技有限公司 | Realize QoS method and apparatus |
| CN106648832A (en) * | 2016-12-09 | 2017-05-10 | 武汉烽火信息集成技术有限公司 | Virtual machine resource utilization improving device and method thereof |
| CN106648832B (en) * | 2016-12-09 | 2020-01-14 | 武汉烽火信息集成技术有限公司 | Device and method for improving resource utilization rate of virtual machine |
| CN107528924A (en) * | 2017-10-09 | 2017-12-29 | 郑州云海信息技术有限公司 | A kind of distributed type assemblies Metadata Service dispositions method and system |
| CN107528924B (en) * | 2017-10-09 | 2021-06-25 | 郑州云海信息技术有限公司 | Distributed cluster metadata service deployment method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102567089B (en) | 2014-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102567089B (en) | Design method for thread pool of metadata server in distributed file system | |
| RU2666245C1 (en) | Method and device for storage of data with high level of multithreading | |
| JP4694595B2 (en) | Sleep queue management | |
| US9197703B2 (en) | System and method to maximize server resource utilization and performance of metadata operations | |
| CN102541659B (en) | Method and device for processing of server service requests | |
| Grover et al. | Data Ingestion in AsterixDB. | |
| US20150154297A1 (en) | Real-time distributed in memory search architecture | |
| WO2016191620A1 (en) | System and method for transparent context aware filtering of data requests | |
| CN103294786A (en) | Metadata organization and management method and system of distributed file system | |
| WO2017028696A1 (en) | Method and device for monitoring load of distributed storage system | |
| JP2005505833A5 (en) | ||
| US10148531B1 (en) | Partitioned performance: adaptive predicted impact | |
| WO2016155238A1 (en) | File reading method in distributed storage system, and server end | |
| CN106909554B (en) | Method and device for loading database text table data | |
| CN102902589A (en) | Method for managing and scheduling cluster MIS (Many Integrated Core) job | |
| Chen et al. | Fpga-accelerated samplesort for large data sets | |
| CN102760073A (en) | Method, system and device for scheduling task | |
| Tang et al. | Mrorder: Flexible job ordering optimization for online mapreduce workloads | |
| KR101830504B1 (en) | In-Memory DB Connection Support Type Scheduling Method and System for Real-Time Big Data Analysis in Distributed Computing Environment | |
| Shu et al. | Dynamic load balancing and channel strategy for apache flume collecting real-time data stream | |
| Hurst et al. | Social streams blog crawler | |
| CN107911484B (en) | Message processing method and device | |
| CN1825288A (en) | Method for implementing process multi-queue dispatching of embedded SRAM operating system | |
| CN102929714A (en) | uC/OS-II-based hardware task manager | |
| US10671636B2 (en) | In-memory DB connection support type scheduling method and system for real-time big data analysis in distributed computing environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160426 Address after: 211106 Jiangning economic and Technological Development Zone, Jiangsu, general Avenue, No. 37, No. Patentee after: JIANGSU DAWN INFORMATION TECHNOLOGY CO., LTD. Address before: 100084 Beijing Haidian District City Mill Street No. 64 Patentee before: Dawning Information Industry (Beijing) Co., Ltd. |