CN101877010A - Distributed file system and associated storage method thereof - Google Patents
Distributed file system and associated storage method thereof Download PDFInfo
- Publication number
- CN101877010A CN101877010A CN2010102034470A CN201010203447A CN101877010A CN 101877010 A CN101877010 A CN 101877010A CN 2010102034470 A CN2010102034470 A CN 2010102034470A CN 201010203447 A CN201010203447 A CN 201010203447A CN 101877010 A CN101877010 A CN 101877010A
- Authority
- CN
- China
- Prior art keywords
- file
- client
- file description
- description structure
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed file system and an associated storage method thereof, wherein the method comprises the following steps of: establishing a file description architecture by a client end, wherein the file description architecture comprises data stream information and is used for storing data stream fields of metadata expanded in advance; and when the client end carries out a reading operation, reading the data stream information of the file description architecture. Through the invention, the reading speed and the integral concurrent processing capacity of the system can be improved.
Description
Technical field
The present invention relates to communication field, relate in particular to a kind of distributed file system and associated storage method thereof.
Background technology
A distributed file system generally comprises a meta data server (Metadata Server) and a plurality of data server (Data Server) and a plurality of client (Client).Meta data server and data server are by client-access, specifically as shown in Figure 1:
File is divided into the piece (CHUNK) of fixed measure, is stored on the Data Server, and in order to guarantee reliability, each CHUNK can be stored on a plurality of Data Server.
The metadata that Metadata Server management document is all comprises file designation space, access control information, and file is to the map information of CHUNK, and the position at the current place of CHUNK etc.It is the activity of management system scope also, and for example CHUNK rents management, the garbage reclamation of orphan CHUNK, and CHUNK moving between Data Server.Metadata Server periodically with each Data Server communication, gives them to indicate and to collect their state with heartbeat message.
Client has realized file system application programming interfaces (Application Program Interface is called for short API), helps application program and Metadata Server and Data Server communication, and data are read and write.Client carries out metadata operation alternately with Metadata Server, and Client directly carries out data manipulation according to these metadata informations and Data Server.
Fig. 2 shows the flow process that Client opens file, and is first steps necessary of file access, comprising: Client sends to Metadata Server and opens (open) file, and Metadata Server returns metadata, comprising data message; Client carries out and creates the file description structure.
Fig. 3 is the flow process that the Client of prior art reads file, with reference to figure 3, when upper level applications is called the read interface, Client is according to reading parameter to position that CHUNK is read in Metadata Server request, this positional information is returned to Client in the success back, directly this CHUNK on the Data Server is carried out read operation by Client at last.
According to above-mentioned processing, have following problem: for a small documents (for example: less than the small documents of a certain preset value MIN) very, this general flow process of reading seems lengthy.Under the application scenario of some special a large amount of storage small documents, system can cause reading decrease in efficiency owing to visiting small documents repeatedly, therefore is necessary to propose improved technological means, solves this problem.
Summary of the invention
Because prior art exists and existingly visits small documents repeatedly and cause reading the problem of decrease in efficiency and make the present invention, fundamental purpose of the present invention for this reason is to provide a kind of distributed file system and associated storage method thereof, wherein:
Associated storage method according to the distributed file system of the embodiment of the invention comprises: client is created the file description structure, comprises traffic flow information in the file description structure, and it is used to store the data stream field of the metadata that expands in advance; Client reads the traffic flow information of file description structure when read operation.
Preferably, before client was carried out read operation, this method further comprised: client is carried out write operation to the traffic flow information of file description structure.
Preferably, before client was created the file description structure, this method further comprises: client was obtained the metadata that comprises the data stream field by meta data server when opening file.
Preferably, client is created the file description structure and is further comprised: the metadata creation file description structure that the client basis gets access to, and backspace file handle.
Preferably, client reads the traffic flow information of file description structure when read operation, and further comprise: client is when read operation, navigate to corresponding file description scheme body according to file handle, and the traffic flow information of file description structure is read.
Distributed file system according to the embodiment of the invention, comprise client and meta data server, wherein client comprises: creation module is used to create the file description structure, comprise traffic flow information in the file description structure, it is used to store the data stream field of the metadata that expands in advance; The read operation module is used for the traffic flow information of file description structure is read.
Preferably, client further comprises: the write operation module is used for the traffic flow information of file description structure is carried out write operation.
Preferably, client further comprises: acquisition module, be used for when opening file, and obtain the metadata that comprises the data stream field by meta data server.
Preferably, creation module is further used for the metadata creation file description structure that gets access to according to acquisition module and backspace file handle.
Preferably, the read operation module is further used for navigating to corresponding file description scheme body according to file handle, and the traffic flow information of file description structure is read.
Compared with prior art,, shortened the flow process that client reads small documents, avoided the visit of meta data server and frequent hard disk operation have been improved the reading speed and the integral concurrent processing capacity of system according to technical scheme of the present invention.
Description of drawings
Accompanying drawing described herein is used to provide further understanding of the present invention, constitutes the application's a part, and illustrative examples of the present invention and explanation thereof are used to explain the present invention, do not constitute improper qualification of the present invention.In the accompanying drawings:
Fig. 1 is an existing distributed file system Organization Chart;
Fig. 2 is the synoptic diagram of the treatment scheme that opens file of existing distributed file system;
Fig. 3 is the synoptic diagram that reads treatment scheme of existing distributed file system;
Fig. 4 is the synoptic diagram according to the metadata extended data stream field of the embodiment of the invention;
Fig. 5 is the process flow diagram according to the associated storage method of the distributed file system of the embodiment of the invention;
Fig. 6 is the synoptic diagram according to the treatment scheme that opens file of the associated storage method of the distributed file system of the embodiment of the invention;
Fig. 7 is the synoptic diagram that writes treatment scheme according to the associated storage method of the distributed file system of the embodiment of the invention;
Fig. 8 is the synoptic diagram that reads treatment scheme according to the associated storage method of the distributed file system of the embodiment of the invention;
Fig. 9 is the block diagram of distributed file system according to an embodiment of the invention.
Embodiment
Main thought of the present invention mainly is, client is created the file description structure in this locality, this document description scheme body comprises a traffic flow information that is used for the storing metadata data stream, for the read operation less than the data stream of preset value, directly the file description structure is read.
For making the purpose, technical solutions and advantages of the present invention clearer,, the present invention is described in further detail below in conjunction with drawings and the specific embodiments.
According to embodiments of the invention, provide a kind of associated storage method of distributed file system.
At first, to data stream attribute of metadata expansion of file, make full use of the storage capacity of meta data server, with reference to figure 4, Fig. 4 is the synoptic diagram according to the metadata extended data stream field of the embodiment of the invention.As shown in Figure 4, the metadata fields attribute of existing file comprises: file name (Filename) file size (File size), timestamp (Time stamp), file type field attribute information such as (File type).In the present invention, metadata fields is expanded, increased a data stream (Data stream) attribute field, this data stream property is used for the data stream of storing metadata.
Fig. 5 is the process flow diagram according to the associated storage method of the distributed file system of the embodiment of the invention, and as shown in Figure 5, this method comprises:
Step S502, client is created the file description structure, comprises traffic flow information in the file description structure, and it is used to store the data stream field of the metadata that expands in advance.
The file description structure is used to store the metadata attributes that client is obtained from server, usually, the file description structure is corresponding one by one with meta-data attribute information, since in server-side extensions a data stream attribute field, so the content of client in obtaining the data stream property field of server end need be created a file description structure, to store this data stream field.
Client is obtained the metadata that comprises the data stream field by meta data server when opening file, and according to the metadata creation file description structure that gets access to, and the backspace file handle.
Step S504, client reads the traffic flow information of file description structure when read operation.Client navigates to corresponding file description scheme body according to file handle.
Through above-mentioned processing, client has realized the read operation of small documents in this locality, avoided the visit of meta data server and frequent hard disk operation, has therefore shortened the flow process that client reads small documents,, reached the purpose that improves data-handling efficiency.
Describe the embodiment of the invention in detail below in conjunction with Fig. 6, Fig. 7 and Fig. 8.
In distributed file system, file data is carried out before the read operation, to open the processing operation of this document earlier.Fig. 6 is the synoptic diagram according to the treatment scheme that opens file of the associated storage method of the distributed file system of the embodiment of the invention, and as shown in Figure 6, this processing may further comprise the steps:
Step S602, application call open interface sends (open file) request that opens file by client (Client) to meta data server (Metadata Server).
Step S604, meta data server is received the metadata that is written into this document after the request of opening, and judge whether contain active traffic in this metadata according to the file size information in the metadata, if data stream is arranged, just the part of message (metadata information) returns to client in response.
Step S606, client is created corresponding file description scheme body (File Description Structure) according to getting access to metadata information (containing data stream), and the backspace file handle is given the application program on upper strata.
After opening file, just can carry out write operation to file.Fig. 7 is the synoptic diagram that writes treatment scheme according to the associated storage method of the distributed file system of the embodiment of the invention, as shown in Figure 7, on the basis of processing shown in Figure 6, also comprises following processing:
Step S702, after upper level applications gets access to file handle, by handle when file writes (write) data, at first navigate to corresponding file description scheme body according to file handle, if the valid data stream information is arranged in this structure, directly write data in the data stream field of file description structure, and revise other metadata fields (size, timestamp etc.) synchronously;
Step S704, if cause data stream size to change (writing such as the formula of appending) in write operation, then client is initiated the file size Notification of Changes to meta data server.
Step S706, meta data server judges according to preset value (MIN) whether current file satisfies the small documents condition, if current file then returns acknowledge message to client less than preset value (MIN) (being that current file still is small documents);
Step S708, client sends to meta data server by the file size updating message with the document data flow that upgrades, so that the document data flow of client and meta data server is consistent.
Need to prove, in step S706, if meta data server judges that current file is greater than preset value (MIN) (being that current file becomes big file), then meta data server notice client need be written to the data stream in the file description structure on the data server, client is initiated traditional flow process that writes afterwards, data stream in the file description structure is write data server, and all write requests are afterwards initiated conventional write go into flow process, concrete treatment step please refer to Fig. 3, does not give unnecessary details herein.
Generally speaking, the user can set in advance the threshold value of metadata store file size according to the size that writes small documents, for example: this small documents that writes size is 10K, it is 10K that MIN then is set, promptly the file less than 10K is a small documents, carry out flow process as shown in Figure 7, the file that surpasses 10K is big file, carries out flow process as shown in Figure 3.Write the value that to reset MIN when handling carrying out file next time, it is noted that the size that the MIN value also needs the taking into account system internal memory is set.
After carrying out write operation, can also comprise read operation to this document.Fig. 8 is the synoptic diagram that reads treatment scheme according to the associated storage method of the distributed file system of the embodiment of the invention.With reference to figure 8, Client is when reading file (read), at first navigate to corresponding file description scheme body according to file handle, if valid data stream information (being that file is the small documents in the threshold values definition) is arranged in this structure, then directly from the file description structure of client, carries out reading of data.Otherwise initiate the operation of Fig. 3.
System embodiment
According to embodiments of the invention, also provide a kind of distributed file system.
Fig. 9 is the block diagram of distributed file system according to an embodiment of the invention, and as shown in Figure 9, this system comprises: client 10 and meta data server 20, and wherein this client 10 comprises:
Acquisition module 110 is used for when opening file, and obtains the metadata that comprises the data stream field by meta data server.Creation module 120 is used for the metadata creation file description structure that gets access to according to acquisition module 110, comprises traffic flow information in the file description structure, and this traffic flow information is used to store the data stream field of the metadata that expands in advance.Write operation module 130 is used for the traffic flow information of file description structure is carried out write operation.Read operation module 140 is used for the traffic flow information of file description structure is carried out read operation.
In actual applications, can not give unnecessary details with reference to figure 5 to Fig. 8 according to the concrete treatment scheme of association's storage system of the distributed file system of the embodiment of the invention herein.
To sum up, the technique scheme according to the present invention, client realized the read operation of small documents in this locality, shortened the flow process that client is read small documents, avoided the visit of meta data server and frequent hard disk operation have been improved the reading speed and the integral concurrent processing capacity of system.
The above is embodiments of the invention only, is not limited to the present invention, and for a person skilled in the art, the present invention can have various changes and variation.Within the spirit and principles in the present invention all, any modification of being done, be equal to replacement, improvement etc., all should be included within the claim scope of the present invention.
Claims (10)
1. the associated storage method of a distributed file system is characterized in that, comprising:
Client is created the file description structure, comprises traffic flow information in the described file description structure, and it is used to store the data stream field of the metadata that expands in advance;
Described client reads the traffic flow information of described file description structure when read operation.
2. method according to claim 1 is characterized in that, before described client was carried out read operation, described method further comprised:
Described client is carried out write operation to the traffic flow information of described file description structure.
3. method according to claim 1 is characterized in that, before described client was created the file description structure, described method further comprised:
Described client is obtained the metadata that comprises the data stream field by meta data server when opening file.
4. method according to claim 3 is characterized in that, described client is created the file description structure and further comprised:
Described client is according to the described metadata creation file description structure that gets access to, and the backspace file handle.
5. method according to claim 4 is characterized in that, described client reads the traffic flow information of described file description structure when read operation, further comprises:
Described client navigates to corresponding file description scheme body according to described file handle when read operation, and the traffic flow information of described file description structure is read.
6. a distributed file system is characterized in that, comprises client and meta data server, and wherein said client comprises:
Creation module is used to create the file description structure, comprises traffic flow information in the described file description structure, and it is used to store the data stream field of the metadata that expands in advance;
The read operation module is used for the traffic flow information of described file description structure is read.
7. system according to claim 5 is characterized in that, described client further comprises:
The write operation module is used for the traffic flow information of described file description structure is carried out write operation.
8. system according to claim 5 is characterized in that, described client further comprises:
Acquisition module is used for when opening file, and obtains the metadata that comprises the data stream field by described meta data server.
9. system according to claim 6 is characterized in that, described creation module is further used for the described metadata creation file description structure that gets access to according to described acquisition module and backspace file handle.
10. system according to claim 7 is characterized in that, described read operation module is further used for navigating to corresponding file description scheme body according to described file handle, and the traffic flow information of described file description structure is read.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102034470A CN101877010A (en) | 2010-06-12 | 2010-06-12 | Distributed file system and associated storage method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102034470A CN101877010A (en) | 2010-06-12 | 2010-06-12 | Distributed file system and associated storage method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101877010A true CN101877010A (en) | 2010-11-03 |
Family
ID=43019566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010102034470A Pending CN101877010A (en) | 2010-06-12 | 2010-06-12 | Distributed file system and associated storage method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101877010A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050114406A1 (en) * | 2003-11-26 | 2005-05-26 | Veritas Operating Corporation | System and method for detecting and storing file content access information within a file system |
| CN101075241A (en) * | 2006-12-26 | 2007-11-21 | 腾讯科技（深圳）有限公司 | Method and system for processing buffer |
| CN101207542A (en) * | 2007-12-12 | 2008-06-25 | 江苏省电力公司 | Method for immediate addressing of multilevel service bus service invocation under SOA frame construction |
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
-
2010
- 2010-06-12 CN CN2010102034470A patent/CN101877010A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050114406A1 (en) * | 2003-11-26 | 2005-05-26 | Veritas Operating Corporation | System and method for detecting and storing file content access information within a file system |
| CN101075241A (en) * | 2006-12-26 | 2007-11-21 | 腾讯科技（深圳）有限公司 | Method and system for processing buffer |
| CN101207542A (en) * | 2007-12-12 | 2008-06-25 | 江苏省电力公司 | Method for immediate addressing of multilevel service bus service invocation under SOA frame construction |
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7844643B2 (en) | Storage management system with integrated continuous data protection and remote copy | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| US9384201B2 (en) | Method of managing data of file system using database management system | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN103530387A (en) | Improved method aimed at small files of HDFS | |
| CN101707633B (en) | Message-oriented middleware persistent message storing method based on file system | |
| CN103838830A (en) | Data management method and system of HBase database | |
| CN101510838B (en) | Method for implementing perdurable data queue | |
| CN113568582B (en) | Data management method, device and storage equipment | |
| CN111198856B (en) | File management method, device, computer equipment and storage medium | |
| US10884926B2 (en) | Method and system for distributed storage using client-side global persistent cache | |
| CN108932286B (en) | Data query method and device | |
| CN109918352B (en) | Memory system and method of storing data | |
| US8296270B2 (en) | Adaptive logging apparatus and method | |
| CN101965499B (en) | Map updating system and map updating program using dynamic cache memory | |
| US20080320062A1 (en) | Method of transferring file system, file system transference program, and file system transference device | |
| CN107798063A (en) | Snap processing method and snapshot processing unit | |
| CN109189724B (en) | Method and device for improving audio and video data storage efficiency of video monitoring system | |
| US8527478B1 (en) | Handling bulk and incremental updates while maintaining consistency | |
| US8595426B2 (en) | Handling commands within a write-once read-many storage device configuration | |
| US20170286442A1 (en) | File system support for file-level ghosting | |
| CN115878625A (en) | Data processing method and device and electronic equipment | |
| JPWO2010001606A1 (en) | Controller, information recording device, access device, information recording system, and information recording method | |
| CN113360095B (en) | Hard disk data management method, device, equipment and medium | |
| CN101877010A (en) | Distributed file system and associated storage method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20101103 |