CN104158858A - Method and system for accessing distributed cluster file system - Google Patents
Method and system for accessing distributed cluster file system Download PDFInfo
- Publication number
- CN104158858A CN104158858A CN201410368101.4A CN201410368101A CN104158858A CN 104158858 A CN104158858 A CN 104158858A CN 201410368101 A CN201410368101 A CN 201410368101A CN 104158858 A CN104158858 A CN 104158858A
- Authority
- CN
- China
- Prior art keywords
- client
- directory
- quota value
- size
- metadata server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000006870 function Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a system for accessing a distributed cluster file system. The method comprises the following steps: the quota value of each client directory is configured by one of clients in advance and the configured quota value of the client directory is sent to and stored in a metadata server; the method also includes the step that when data is written in the client, the metadata server is informed; based on the information sent by the client, the metadata server compares the size of the occupied space of the client directory with the size of the quota value of the client directory and the client is informed of accepting the data writing only when the size of the occupied space of the client directory is smaller than the size of the quota value of the client directory. The technical scheme of the application has the function of directory quota of the distributed cluster storage system and can remind the client of data processing for the written data according to the quota value of the configured client directory, so as to avoid data loss.
Description
Technical Field
The present invention relates to distributed cluster technologies, and in particular, to a method and an apparatus for accessing a distributed cluster file system.
Background
Currently, PB-class cluster storage systems are already in formal operation in some industries, and a large number of users with different operating system types access the cluster storage systems, which may place demands on client directory capacity. With regard to application of the directory quota, technology for setting quota based on a disk by using a hard disk of a server is mature, but the existing technology is an alternative scheme which cannot realize the directory quota function of the distributed cluster storage system, and can only use the user quota to limit the access space of the user in many cases, so that the user can store data with a certain capacity when storing the data, but can take a long time to find the data when accessing the data. Therefore, how to implement the function of the directory quota of the distributed cluster file system is a technical problem which needs to be solved urgently.
Disclosure of Invention
In order to solve the technical problem, the invention provides a method and a system for accessing a distributed cluster file system, which can realize the directory quota function of the distributed cluster storage system, so that a client can conveniently and quickly access the distributed cluster storage system.
In order to achieve the above object, the present invention discloses a method for accessing a distributed cluster file system, comprising the following steps: one client pre-configures each client directory quota value, sends the configured client directory quota value to a metadata server and stores the client directory quota value in the metadata server; further comprising:
when the client has data to write, informing the metadata server;
and the metadata server compares the size of the used space of the client directory with the size of the quota value of the client directory, and informs the client to receive the data writing when the size of the used space of the client directory is smaller than the quota value of the client directory.
Further, when the size of the used space of the client directory is larger than or equal to the client directory quota value through comparison, the metadata server informs the client to reject the data writing.
Further, before comparing the size of the used space of the client directory with the quota value of the client directory, the method further comprises:
the metadata server obtains the size of the used space of the client directory.
Further, the system of the client is a Linux operating system or a Windows operating system.
The invention also discloses a system for accessing the distributed cluster file system, which comprises: one or more clients and metadata servers; wherein,
each client is used for notifying the metadata server when data is written;
one of the clients is also used for pre-configuring the quota value of each client directory, sending the configured client directory quota value to the metadata server and storing the client directory quota value in the metadata server;
and the metadata server is used for comparing the size of the used space of the client directory with the size of the quota value of the client directory, and informing the client server of receiving the data writing when the size of the used space of the client directory is smaller than the quota value of the client directory.
Further, the metadata server is further configured to:
and when the size of the used space of the client directory is larger than or equal to the client directory quota value through comparison, the client is informed to refuse the data writing.
And further, the metadata server is also used for acquiring the size of the used space of the client directory.
Further, the system of the client is a Linux operating system or a Windows operating system.
The technical scheme of the application includes: one client pre-configures each client directory quota value, sends the configured client directory quota value to a metadata server and stores the client directory quota value in the metadata server; further comprising: when the client has data to write, informing the metadata server; and the metadata server compares the size of the used space of the client directory with the size of the quota value of the client directory according to the notification from the client, and notifies the client to receive the data writing when the size of the used space of the client directory is smaller than the quota value of the client directory. According to the technical scheme, the directory quota function of the distributed cluster storage system is achieved, the client is prompted to process the written data according to the configured client directory quota value, and data loss is avoided.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of a method of accessing a distributed cluster file system of the present invention;
FIG. 2 is a schematic structural diagram of a system for accessing a distributed cluster file system according to the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and the specific embodiments.
FIG. 1 is a flowchart of a method for accessing a distributed cluster file system according to the present invention, as shown in FIG. 1, including the following steps:
step 101, one of the clients pre-configures each client directory quota value, sends the configured client directory quota value to the metadata server, and stores the client directory quota value in the metadata server.
It should be noted that the configured client directory quota value may be stored in a data structure inode in the metadata server.
The directory quota value refers to an upper limit value of storage capacity allocated by the client in the distributed cluster file system. Regarding configuring the directory quota value of each client, one administrator client is available in all clients, and can acquire information of other clients, and the directory quota value is set on all clients (including itself) at the administrator client according to the actual application condition of each client, for example, the administrator client may manage all clients, and those skilled in the art know how to set the directory quota value, and details are not described here.
It should be noted that, when data is written into a file in the distributed cluster file system by a client, basic attributes such as a creator of the file, a creation date of the file, a modification date of the file, and a size of the file are automatically saved or updated in the metadata server, and these information are all stored in corresponding inodes, and the administrator client sets in advance a value of each client directory quota according to information of each client.
Step 102, when the client has data to write, the metadata server is notified.
It should be noted that in this step, when the client has data to write, the client needs to establish message communication with the metadata server, and after the establishment, the client sends a notification to the metadata server.
And 103, comparing the used space size of the client directory with the quota value of the client directory by the metadata server, and notifying the client to accept data writing if the used space size of the client directory is smaller than the quota value of the client directory.
The method further comprises the following steps: and when the size of the used space of the client directory is larger than or equal to the quota value of the client directory, the metadata server informs the client to reject the data writing.
Prior to step 103, the method of the present invention further comprises: the metadata server obtains the size of the used space of the client directory.
In the method, the system of the client is a Linux operating system or a Windows operating system.
It should be noted that, when the client directory quota is full, the client may apply for reconfiguring the client directory quota value, which does not affect the above-mentioned method idea of the present invention.
The method of the invention realizes the directory quota function of the distributed cluster storage system by configuring the client directory quota value for each client and sending the configured quota value to the metadata server, and prompts the client to process the written data according to the configured client directory quota value, thereby avoiding the loss of data.
Fig. 2 is a schematic structural diagram of the system for accessing a distributed cluster file system according to the present invention, as shown in fig. 2, including: one or more clients and a metadata server. Wherein,
each client is used for notifying the metadata server when data is written.
And one of the clients is also used for pre-configuring the quota value of each client directory, sending the configured client directory quota value to the metadata server and storing the client directory quota value in the metadata server.
It should be noted that the configured client directory quota value is stored in a data structure (inode) of the metadata server.
And the metadata server is used for comparing the size of the used space of the client directory with the size of the quota value of the client directory according to the notification from the client, and notifying the client to accept data writing if the size of the used space of the client directory is smaller than the quota value of the client directory.
Further, the metadata server is further configured to: and when the size of the used space of the client directory is larger than or equal to the client directory quota value through comparison, the client is informed to refuse the data writing.
And further, the metadata server is also used for acquiring the size of the used space of the client directory.
The system of the client is a Linux operating system or a Windows operating system.
It will be understood by those skilled in the art that all or part of the steps of the above methods may be implemented by instructing the relevant hardware through a program, and the program may be stored in a computer readable storage medium, such as a read-only memory, a magnetic or optical disk, and the like. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits. Accordingly, each module/unit in the above embodiments may be implemented in the form of hardware, and may also be implemented in the form of a software functional module. The present application is not limited to any specific form of hardware or software combination.
The above description is only a preferred example of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method for accessing a distributed cluster file system is characterized in that one client pre-configures each client directory quota value, sends the configured client directory quota value to a metadata server and stores the client directory quota value in the metadata server; further comprising:
when the client has data to write, informing the metadata server;
and the metadata server compares the size of the used space of the client directory with the size of the quota value of the client directory, and informs the client to receive the data writing when the size of the used space of the client directory is smaller than the quota value of the client directory.
2. The method of claim 1, wherein the metadata server notifies the client of a rejection of the data write when the comparison indicates that the size of the used space of the client directory is greater than or equal to the client directory quota value.
3. The method of claim 1 or 2, wherein prior to the comparing the client directory used space size to the client directory quota value, the method further comprises:
and the metadata server acquires the size of the used space of the client directory.
4. The method according to claim 1 or 2, wherein the system of the client is Linux operating system or Windows operating system.
5. An access distributed cluster file system, comprising: one or more clients and metadata servers; wherein,
each client is used for notifying the metadata server when data is written;
one of the clients is also used for pre-configuring the quota value of each client directory, sending the configured client directory quota value to the metadata server and storing the client directory quota value in the metadata server;
and the metadata server is used for comparing the size of the used space of the client directory with the size of the quota value of the client directory, and informing the client server of receiving the data writing when the size of the used space of the client directory is smaller than the quota value of the client directory.
6. The apparatus of claim 5, wherein the metadata server is further configured to:
and when the size of the used space of the client directory is larger than or equal to the client directory quota value through comparison, the client is informed to refuse the data writing.
7. The system according to claim 5 or 6, wherein said metadata server is further configured to obtain a size of said client directory used space.
8. The system according to claim 5 or 6, wherein the system of the client is Linux operating system or Windows operating system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410368101.4A CN104158858A (en) | 2014-07-29 | 2014-07-29 | Method and system for accessing distributed cluster file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410368101.4A CN104158858A (en) | 2014-07-29 | 2014-07-29 | Method and system for accessing distributed cluster file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104158858A true CN104158858A (en) | 2014-11-19 |
Family
ID=51884263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410368101.4A Pending CN104158858A (en) | 2014-07-29 | 2014-07-29 | Method and system for accessing distributed cluster file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104158858A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104994164A (en) * | 2015-07-08 | 2015-10-21 | 浪潮(北京)电子信息产业有限公司 | Method and device for statistics of catalog information |
| CN106210026A (en) * | 2016-07-05 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The control method of distributed file system multi-client catalogue quota precision and system |
| CN106484834A (en) * | 2016-09-30 | 2017-03-08 | 郑州云海信息技术有限公司 | Distributed file system user's quota data store method, system and server |
| CN106815232A (en) * | 2015-11-27 | 2017-06-09 | 华为技术有限公司 | Catalog management method, apparatus and system |
| CN107506440A (en) * | 2017-08-24 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of quota method and device of distributed system catalogue |
| CN107562386A (en) * | 2017-09-13 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method for writing data and device of distributed storage cluster |
| CN108073723A (en) * | 2018-01-03 | 2018-05-25 | 郑州云海信息技术有限公司 | A kind of file in distributed type assemblies storage is from compressing method and equipment |
| CN108196787A (en) * | 2017-12-21 | 2018-06-22 | 北京鲸鲨软件科技有限公司 | The quota management method and cluster storage system of cluster storage system |
| CN111176569A (en) * | 2019-12-27 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Flow quota method, device, system and equipment and readable storage medium |
| CN111181791A (en) * | 2019-12-31 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Quota management method, device, equipment and storage medium |
| CN111309686A (en) * | 2020-02-20 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Method, device, equipment and medium for controlling consistency of directory quota |
| CN113076063A (en) * | 2021-04-01 | 2021-07-06 | 山东英信计算机技术有限公司 | Directory quota pre-application method, system, storage medium and equipment |
| CN115242818A (en) * | 2022-07-22 | 2022-10-25 | 济南浪潮数据技术有限公司 | Quota management method, device, equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7496609B2 (en) * | 2005-09-01 | 2009-02-24 | Microsoft Corporation | Dirty shutdown recovery of file system filters |
| CN102447736A (en) * | 2011-11-17 | 2012-05-09 | 华中科技大学 | Management and control method for user quota in multi-network storage system |
| CN102523285A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Storage caching method of object-based distributed file system |
| CN103064639A (en) * | 2012-12-28 | 2013-04-24 | 华为技术有限公司 | Method and device for storing data |
| CN103078898A (en) * | 2012-12-18 | 2013-05-01 | 华为技术有限公司 | File system, interface service device and data storage service provision method |
-
2014
- 2014-07-29 CN CN201410368101.4A patent/CN104158858A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7496609B2 (en) * | 2005-09-01 | 2009-02-24 | Microsoft Corporation | Dirty shutdown recovery of file system filters |
| CN102447736A (en) * | 2011-11-17 | 2012-05-09 | 华中科技大学 | Management and control method for user quota in multi-network storage system |
| CN102523285A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Storage caching method of object-based distributed file system |
| CN103078898A (en) * | 2012-12-18 | 2013-05-01 | 华为技术有限公司 | File system, interface service device and data storage service provision method |
| CN103064639A (en) * | 2012-12-28 | 2013-04-24 | 华为技术有限公司 | Method and device for storing data |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104994164A (en) * | 2015-07-08 | 2015-10-21 | 浪潮(北京)电子信息产业有限公司 | Method and device for statistics of catalog information |
| CN106815232A (en) * | 2015-11-27 | 2017-06-09 | 华为技术有限公司 | Catalog management method, apparatus and system |
| CN106210026B (en) * | 2016-07-05 | 2019-06-18 | 浪潮(北京)电子信息产业有限公司 | The control method and system of distributed file system multi-client catalogue quota precision |
| CN106210026A (en) * | 2016-07-05 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The control method of distributed file system multi-client catalogue quota precision and system |
| CN106484834A (en) * | 2016-09-30 | 2017-03-08 | 郑州云海信息技术有限公司 | Distributed file system user's quota data store method, system and server |
| CN106484834B (en) * | 2016-09-30 | 2020-02-07 | 苏州浪潮智能科技有限公司 | Distributed file system user quota data saving method, system and server |
| CN107506440A (en) * | 2017-08-24 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of quota method and device of distributed system catalogue |
| CN107562386A (en) * | 2017-09-13 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method for writing data and device of distributed storage cluster |
| CN108196787A (en) * | 2017-12-21 | 2018-06-22 | 北京鲸鲨软件科技有限公司 | The quota management method and cluster storage system of cluster storage system |
| CN108073723A (en) * | 2018-01-03 | 2018-05-25 | 郑州云海信息技术有限公司 | A kind of file in distributed type assemblies storage is from compressing method and equipment |
| CN111176569A (en) * | 2019-12-27 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Flow quota method, device, system and equipment and readable storage medium |
| CN111181791A (en) * | 2019-12-31 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Quota management method, device, equipment and storage medium |
| CN111309686A (en) * | 2020-02-20 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Method, device, equipment and medium for controlling consistency of directory quota |
| CN113076063A (en) * | 2021-04-01 | 2021-07-06 | 山东英信计算机技术有限公司 | Directory quota pre-application method, system, storage medium and equipment |
| CN113076063B (en) * | 2021-04-01 | 2023-01-06 | 山东英信计算机技术有限公司 | Directory quota pre-application method, system, storage medium and equipment |
| CN115242818A (en) * | 2022-07-22 | 2022-10-25 | 济南浪潮数据技术有限公司 | Quota management method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104158858A (en) | Method and system for accessing distributed cluster file system | |
| US11886707B2 (en) | Dataset space reclamation | |
| US9658781B2 (en) | Tier based data file management | |
| WO2015117426A1 (en) | File management method and device | |
| CN109660372B (en) | SDN service configuration method and device | |
| US20150227605A1 (en) | Information processing terminal, synchronization control method, and computer-readable recording medium | |
| US10516732B2 (en) | Disconnected ingest in a distributed storage system | |
| CN110941393A (en) | Logical volume management-based LV supply method, device, equipment and medium | |
| CN113626286A (en) | Multi-cluster instance processing method and device, electronic equipment and storage medium | |
| CN104519078A (en) | A method and a device for realizing cloud storage | |
| CN105095103A (en) | Storage device management method and device used for cloud environment | |
| US20190007359A1 (en) | Mail information management method and associated apparatus | |
| US8966238B2 (en) | Personalization of shared electronic devices | |
| US10803109B2 (en) | Method and device for reading and writing video data in NAS device | |
| US11252121B2 (en) | Message sending method and terminal device | |
| US11176089B2 (en) | Systems and methods for implementing dynamic file systems | |
| CN105389316A (en) | File management system and method | |
| CN104461709B (en) | The control method and device of task scheduling | |
| EP2960778A1 (en) | A storage abstraction layer and a system and a method thereof | |
| US20160103849A1 (en) | Heterogeneous storing server and file storing method thereof | |
| CN105760431A (en) | Method and device for transferring file blocks | |
| US20200244525A1 (en) | System and method for device configuration update | |
| US10938919B1 (en) | Registering client devices with backup servers using domain name service records | |
| CN104461382A (en) | Internal writing method for file server operating multiple file systems and server | |
| CN115550317B (en) | Network resource management method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141119 |
|
| RJ01 | Rejection of invention patent application after publication |