CN103761275A - Management method for metadata in distributed file system - Google Patents
Management method for metadata in distributed file system Download PDFInfo
- Publication number
- CN103761275A CN103761275A CN201410009764.7A CN201410009764A CN103761275A CN 103761275 A CN103761275 A CN 103761275A CN 201410009764 A CN201410009764 A CN 201410009764A CN 103761275 A CN103761275 A CN 103761275A
- Authority
- CN
- China
- Prior art keywords
- metadata
- data server
- meta
- meta data
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000004321 preservation Methods 0.000 claims description 11
- 108091027981 Response element Proteins 0.000 claims description 4
- 239000012141 concentrate Substances 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 241001475178 Dira Species 0.000 description 3
- LXMSZDCAJNLERA-ZHYRCANASA-N spironolactone Chemical compound C([C@@H]1[C@]2(C)CC[C@@H]3[C@@]4(C)CCC(=O)C=C4C[C@H]([C@@H]13)SC(=O)C)C[C@@]21CCC(=O)O1 LXMSZDCAJNLERA-ZHYRCANASA-N 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- RMCCONIRBZIDTH-UHFFFAOYSA-N 2-(2-methylprop-2-enoyloxy)ethyl 1,3-dioxo-2-benzofuran-5-carboxylate Chemical compound CC(=C)C(=O)OCCOC(=O)C1=CC=C2C(=O)OC(=O)C2=C1 RMCCONIRBZIDTH-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention provides a management method for metadata in a distributed file system. The method includes the special management processes that the system is divided into a metadata server providing metadata services and a data server, and a user can acquire a file system view through the metadata sever to obtain a directory structure and a file list of the file system; when the user visits a metadata cluster, if the metadata to be accessed are not found in the metadata cluster, data are provided to the user through the data server; when a memory occupied by a metadata cache exceeds a threshold value, part of the memory is released so as to control memory occupancy of the metadata server. Compared with the prior art, the management method for the metadata in the distributed file system has the advantages that single-point problems are solved by the metadata by the aid of a cluster mode, the metadata are stored in the data server, further the system is started rapidly, and system availability is improved.
Description
Technical field
The present invention relates to the Technology On Data Encryption of cluster computer, more specifically say the management method of metadata in distributed file system.
Background technology
Along with development, cloud computing and the arrival of data age greatly of infotech, people need to process original more data, generally by distributed file system, store the data of magnanimity, distributed file system exists various problems at present, such as the Single Point of Faliure of meta data server, this problem generally solves by two-node cluster hot backup.In order to reduce load, the general method of using metadata and data separating, metadata spanned file is persisted to local disk, the start-up time of system is by linear with the scale of system like this, when file system scale exceedes 5,000 ten thousand, to reach dozens of minutes start-up time, the time that system is recovered from fault is long especially, have a strong impact on the availability of system, when meta data file damages accidentally, system will be thoroughly unavailable, need to take the technological means of various complexity to improve the security of meta data file, but meta data file still has the risk of loss.
Based on this, need a kind of management method that can effectively solve metadata loss, the metadata in distributed file system.
Summary of the invention
Technical assignment of the present invention is to solve the deficiencies in the prior art, and the management method of metadata in a kind of distributed file system is provided.
Technical scheme of the present invention realizes in the following manner, the management method of metadata in this kind of distributed file system, and its concrete management process is:
One, first system is divided into two parts: meta data server and data server, wherein meta data server provides Metadata Service, and user can obtain file system view by meta data server, obtains bibliographic structure and the listed files of file system; Data server provides data, services, and user's read-write operation provides service by data server;
Two, when system starts, by data server, converge and obtain metadata, data server is reported a small amount of metadata simultaneously, these metadata caches are on meta data server, a meta data server cluster of many machine compositions, metadata in metadata cluster has been preserved a part of metadata separately, and these metadata are not identical;
Three, when user's accesses meta-data cluster, if do not find the metadata that will access in metadata cluster, meta data server just sends broadcast to data server, when data server is received broadcast request, the state of inquiring about oneself, returns to meta data server by data, then returns to user, and be buffered in the internal memory of meta data server, for access next time;
When the internal memory four, taking when metadata cache exceeds threshold value, partial memory is discharged, to control the EMS memory occupation of meta data server.
Save data and metadata on described data server, in the extended attribute of meta-data preservation in local file system, when data server starts, sweep test metadata sends to meta data server, completion system starts, and then starts thread, the disk of scanning management, by meta-data preservation in internal memory, with the broadcast request of quick response element data server.
The detailed process of described step 2 is: when data server starts, managed disk is carried out to range traversal, only travel through two-stage catalogue, finish traversal, traversing result is sent to the startup that meta data server is completion system, if file directory level is more shallow, all concentrate on a two-stage, travel through the regular hour stop traversal, on the meta data server that metadata is sent, with the start-up time of control system, metadata arrives after meta data server cluster, and metadata cluster deposits and share the storage of bibliographic structure to metadata groupings.
When the internal memory that the metadata of the meta data server buffer memory in described step 4 takies exceeds threshold value, the internal memory that meta data server discharges is the metadata of least commonly using.
The beneficial effect that the present invention compared with prior art produced is:
Metadata in a kind of distributed file system of the present invention in the management method of metadata is used trunking mode to solve single-point problem, meta-data preservation is on data server, system converges to metadata on meta data server while starting, and scale is converged in control, to start rapidly system, improve system availability, together with metadata is kept at data, can greatly strengthen the security of metadata; In the extended attribute of meta-data preservation in local file system, together with data coupling, metadata no longer includes the risk of loss, guarantees consistance and the security of data, practical, is easy to promote.
Accompanying drawing explanation
Accompanying drawing 1 is the structural representation of the embodiment of the present invention.
Embodiment
Below in conjunction with accompanying drawing, the management method of metadata in a kind of distributed file system of the present invention is described in detail below.
The invention provides the management method of metadata in a kind of distributed file system, its concrete management process is:
One, first system is divided into two parts: meta data server and data server, wherein meta data server provides Metadata Service, and user can obtain file system view by meta data server, obtains bibliographic structure and the listed files of file system; Data server provides data, services, and user's read-write operation provides service by data server;
Two, when system starts, by data server, converge and obtain metadata, data server is reported a small amount of metadata simultaneously, these metadata caches are on meta data server, a meta data server cluster of many machine compositions, metadata in metadata cluster has been preserved a part of metadata separately, and these metadata are not identical;
Three, when user's accesses meta-data cluster, if do not find the metadata that will access in metadata cluster, meta data server just sends broadcast to data server, when data server is received broadcast request, the state of inquiring about oneself, returns to meta data server by data, then returns to user, and be buffered in the internal memory of meta data server, for access next time;
When the internal memory four, taking when metadata cache exceeds threshold value, partial memory is discharged, to control the EMS memory occupation of meta data server.
The structure of invention is take metadata cluster as core, and in metadata cluster, each meta data server has been preserved a part of metadata, does mutually mirror image between meta data server, to tackle the fault of meta data server.
System is Preservation Metadata file not, metadata is by dynamically generating, when user uses file system, can not access all files, according to principle of locality, meta data server there is no need to preserve all metadata, as long as preserve part, uses, when the metadata of access does not exist, by broadcasting to data server request.Data server Preservation Metadata, metadata is shared on all data servers, system loading is shared on all nodes, system there is better retractility.
When data server starts, sweep test metadata, sends on meta data server the quick startup of completion system, then start thread, the disk that scanning is managed, reads metadata information in internal memory, with the broadcast request of quick response element data server.Meta-data preservation is in the extended attribute of local file system, together with deposit data, to guarantee consistance and the security of data.
Embodiment: as shown in Figure 1, the entirety of this file system forms meta data server cluster by 4 meta data servers and 5 data servers form.Metadata cluster has been shared the meta-data preservation work of dirA and dirB, and backups each other between two.
5 data servers have respectively been stored 100,000 files, when data server starts, managed disk is carried out to range traversal, only travel through two-stage catalogue, finish traversal, traversing result is sent to the startup that meta data server is completion system, if file directory level is more shallow, all concentrate on a two-stage, travel through the regular hour stop traversal, on the meta data server that metadata is sent, with the start-up time of control system.
Metadata arrives after meta data server cluster, metadata cluster is deposited metadata groupings, such as having two catalogue dirA and dirB, meta data server 1 and 2 is deposited dirA, meta data server 3 and 4 is deposited dirB, 1 and 2,3 and 4 backup each other, to tackle the damage of metadata.Meta data server cluster is shared the storage of bibliographic structure, effectively reduces the load of single meta data server, can support more massive application.When the metadata of user's access does not exist in meta data server, meta data server sends broadcast to data server, request metadata, and data server returns to metadata, then returns to user.
After data server starts, start thread, the disk that scanning is managed, meta-data preservation, in the extended attribute of local file system, is read out and is then kept in internal memory, with the broadcast request of quick response element data server.
User understands the state of metadata cluster, know metadata cluster how many nodes consists of, each node is safeguarded the metadata of which catalogue, mirror between node, thereby ask suitable meta data server, and can automatically switch in the meta data server of mirror image when the meta data server of request breaks down.
The internal memory taking when the metadata of meta data server buffer memory exceeds after threshold value, and meta data server discharges least conventional metadata, to control the use to internal memory.
The foregoing is only embodiments of the invention, within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (4)
1. a management method for metadata in distributed file system, is characterized in that its concrete management process is:
One, first system is divided into two parts: meta data server and data server, wherein meta data server provides Metadata Service, and user can obtain file system view by meta data server, obtains bibliographic structure and the listed files of file system; Data server provides data, services, and user's read-write operation provides service by data server;
Two, when system starts, by data server, converge and obtain metadata, data server is reported a small amount of metadata simultaneously, these metadata caches are on meta data server, a meta data server cluster of many machine compositions, metadata in metadata cluster has been preserved a part of metadata separately, and these metadata are not identical;
Three, when user's accesses meta-data cluster, if do not find the metadata that will access in metadata cluster, meta data server just sends broadcast to data server, when data server is received broadcast request, the state of inquiring about oneself, returns to meta data server by data, then returns to user, and be buffered in the internal memory of meta data server, for access next time;
When the internal memory four, taking when metadata cache exceeds threshold value, partial memory is discharged, to control the EMS memory occupation of meta data server.
2. the management method of metadata in a kind of distributed file system according to claim 1, it is characterized in that: save data and metadata on described data server, in the extended attribute of meta-data preservation in local file system, when data server starts, sweep test metadata sends to meta data server, and completion system starts, then start thread, the disk of scanning management, by meta-data preservation in internal memory, with the broadcast request of quick response element data server.
3. the management method of metadata in a kind of distributed file system according to claim 2, it is characterized in that: the detailed process of described step 2 is: when data server starts, managed disk is carried out to range traversal, only travel through two-stage catalogue, finish traversal, traversing result is sent to the startup that meta data server is completion system, if file directory level is more shallow, all concentrate on a two-stage, travel through the regular hour stop traversal, on the meta data server that metadata is sent, with the start-up time of control system, metadata arrives after meta data server cluster, metadata cluster deposits and shares the storage of bibliographic structure to metadata groupings.
4. the management method of metadata in a kind of distributed file system according to claim 2, it is characterized in that: when the internal memory that the metadata of the meta data server buffer memory in described step 4 takies exceeds threshold value, the internal memory that meta data server discharges is the metadata of least commonly using.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410009764.7A CN103761275B (en) | 2014-01-09 | 2014-01-09 | The management method of metadata in a kind of distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410009764.7A CN103761275B (en) | 2014-01-09 | 2014-01-09 | The management method of metadata in a kind of distributed file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103761275A true CN103761275A (en) | 2014-04-30 |
| CN103761275B CN103761275B (en) | 2018-04-13 |
Family
ID=50528512
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410009764.7A Active CN103761275B (en) | 2014-01-09 | 2014-01-09 | The management method of metadata in a kind of distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103761275B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095495A (en) * | 2015-08-21 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Distributed file system cache management method and system |
| CN105426126A (en) * | 2015-11-11 | 2016-03-23 | 浪潮(北京)电子信息产业有限公司 | Method and device for constructing multi-channel constant rate IO (Input Output) cache of cloud storage client |
| CN107015989A (en) * | 2016-01-27 | 2017-08-04 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device |
| CN107633070A (en) * | 2017-09-22 | 2018-01-26 | 郑州云海信息技术有限公司 | Balance Control Scheme method, apparatus and storage medium without the MDS of configuration |
| CN107665253A (en) * | 2017-09-22 | 2018-02-06 | 郑州云海信息技术有限公司 | Configurable MDS balance control method, device and storage medium |
| CN108287664A (en) * | 2018-01-02 | 2018-07-17 | 江苏科海智能系统有限公司 | A kind of fast large based on NVM storage devices is according to system and its design method |
| CN109189480A (en) * | 2018-07-02 | 2019-01-11 | 新华三技术有限公司成都分公司 | File system starts method and device |
| CN109582649A (en) * | 2018-12-03 | 2019-04-05 | 郑州云海信息技术有限公司 | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing |
| CN109583222A (en) * | 2018-12-07 | 2019-04-05 | 浪潮(北京)电子信息产业有限公司 | Distributed file system metadata service device recycles client jurisdiction method and device |
| CN109976667A (en) * | 2016-01-26 | 2019-07-05 | 华为技术有限公司 | A kind of mirror image management method, apparatus and system |
| CN110263061A (en) * | 2019-06-17 | 2019-09-20 | 郑州阿帕斯科技有限公司 | A kind of data query method and system |
| CN111078593A (en) * | 2019-12-29 | 2020-04-28 | 浪潮电子信息产业股份有限公司 | Object metadata access method, system and related device |
| CN114153374A (en) * | 2021-08-04 | 2022-03-08 | 北京天德科技有限公司 | Distributed storage system for storing metadata and data together |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| CN101997884A (en) * | 2009-08-18 | 2011-03-30 | 升东网络科技发展(上海)有限公司 | Distributed storage system and method |
| US20110191691A1 (en) * | 2010-01-29 | 2011-08-04 | Spears Joseph L | Systems and Methods for Dynamic Generation and Management of Ancillary Media Content Alternatives in Content Management Systems |
| CN102546751A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Hierarchical metadata cache control method of distributed file system |
-
2014
- 2014-01-09 CN CN201410009764.7A patent/CN103761275B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101997884A (en) * | 2009-08-18 | 2011-03-30 | 升东网络科技发展(上海)有限公司 | Distributed storage system and method |
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| US20110191691A1 (en) * | 2010-01-29 | 2011-08-04 | Spears Joseph L | Systems and Methods for Dynamic Generation and Management of Ancillary Media Content Alternatives in Content Management Systems |
| CN102546751A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Hierarchical metadata cache control method of distributed file system |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095495B (en) * | 2015-08-21 | 2019-01-25 | 浪潮(北京)电子信息产业有限公司 | A kind of distributed file system buffer memory management method and system |
| CN105095495A (en) * | 2015-08-21 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Distributed file system cache management method and system |
| CN105426126A (en) * | 2015-11-11 | 2016-03-23 | 浪潮(北京)电子信息产业有限公司 | Method and device for constructing multi-channel constant rate IO (Input Output) cache of cloud storage client |
| CN105426126B (en) * | 2015-11-11 | 2018-06-05 | 浪潮(北京)电子信息产业有限公司 | The construction method and device of cloud storage client multichannel constant rate of speed IO cachings |
| CN109976667A (en) * | 2016-01-26 | 2019-07-05 | 华为技术有限公司 | A kind of mirror image management method, apparatus and system |
| CN109976667B (en) * | 2016-01-26 | 2021-09-14 | 华为技术有限公司 | Mirror image management method, device and system |
| CN107015989A (en) * | 2016-01-27 | 2017-08-04 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device |
| CN107633070A (en) * | 2017-09-22 | 2018-01-26 | 郑州云海信息技术有限公司 | Balance Control Scheme method, apparatus and storage medium without the MDS of configuration |
| CN107665253A (en) * | 2017-09-22 | 2018-02-06 | 郑州云海信息技术有限公司 | Configurable MDS balance control method, device and storage medium |
| CN107665253B (en) * | 2017-09-22 | 2022-02-18 | 郑州云海信息技术有限公司 | Configurable MDS balance control method and device and storage medium |
| CN108287664A (en) * | 2018-01-02 | 2018-07-17 | 江苏科海智能系统有限公司 | A kind of fast large based on NVM storage devices is according to system and its design method |
| CN109189480B (en) * | 2018-07-02 | 2021-11-09 | 新华三技术有限公司成都分公司 | File system starting method and device |
| CN109189480A (en) * | 2018-07-02 | 2019-01-11 | 新华三技术有限公司成都分公司 | File system starts method and device |
| CN109582649A (en) * | 2018-12-03 | 2019-04-05 | 郑州云海信息技术有限公司 | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing |
| CN109582649B (en) * | 2018-12-03 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata storage method, device and equipment and readable storage medium |
| CN109583222A (en) * | 2018-12-07 | 2019-04-05 | 浪潮(北京)电子信息产业有限公司 | Distributed file system metadata service device recycles client jurisdiction method and device |
| CN110263061A (en) * | 2019-06-17 | 2019-09-20 | 郑州阿帕斯科技有限公司 | A kind of data query method and system |
| CN111078593A (en) * | 2019-12-29 | 2020-04-28 | 浪潮电子信息产业股份有限公司 | Object metadata access method, system and related device |
| CN111078593B (en) * | 2019-12-29 | 2022-08-19 | 浪潮电子信息产业股份有限公司 | Object metadata access method, system and related device |
| CN114153374A (en) * | 2021-08-04 | 2022-03-08 | 北京天德科技有限公司 | Distributed storage system for storing metadata and data together |
| CN114153374B (en) * | 2021-08-04 | 2022-06-28 | 北京天德科技有限公司 | Distributed storage system for jointly storing metadata and data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103761275B (en) | 2018-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103761275A (en) | Management method for metadata in distributed file system | |
| US11301154B2 (en) | Distributed storage method and device | |
| CN107169083B (en) | Mass vehicle data storage and retrieval method and device for public security card port and electronic equipment | |
| WO2017167171A1 (en) | Data operation method, server, and storage system | |
| EP2863310B1 (en) | Data processing method and apparatus, and shared storage device | |
| US20160212206A1 (en) | Deterministic database system and data transferring method thereof | |
| US11245774B2 (en) | Cache storage for streaming data | |
| US10191915B2 (en) | Information processing system and data synchronization control scheme thereof | |
| CN104243425A (en) | Content management method, device and system in content delivery network | |
| CN106202112A (en) | CACHE DIRECTORY method for refreshing and device | |
| CN103518364A (en) | Data update method for distributed storage system and server | |
| CN102710763B (en) | The method and system of a kind of distributed caching pond, burst and Failure Transfer | |
| US9262511B2 (en) | System and method for indexing streams containing unstructured text data | |
| CN105159845A (en) | Memory reading method | |
| US20120278429A1 (en) | Cluster system, synchronization controlling method, server, and synchronization controlling program | |
| WO2016095149A1 (en) | Data compression and storage method and device, and distributed file system | |
| CN106326239A (en) | Distributed file system and file meta-information management method thereof | |
| CN104516974A (en) | Management method and device for file system directory entry | |
| CN108540510B (en) | Cloud host creation method and device and cloud service system | |
| US20230409527A1 (en) | Method And System For Deleting Obsolete Files From A File System | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| CN102984256A (en) | Processing method and system for metadata based on authorization manner | |
| CN103365987A (en) | Clustered database system and data processing method based on shared-disk framework | |
| CN109165078B (en) | Virtual distributed server and access method thereof | |
| JP2012018607A (en) | Distributed cache system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180313 Address after: 510640 Guangdong city of Guangzhou province Whampoa Tianhe District Road No. 163 Xiping Yunlu 9 layer A tower by 01 unit Applicant after: Guangdong Inspur Big Data Research Co., Ltd. Address before: 250014 Shandong Province, Ji'nan City hi tech Development Zone, Nga Road No. 1036 Applicant before: Langchao Electronic Information Industry Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |