CN109344122A - A kind of distributed meta-data management method and system for creating strategy in advance based on file - Google Patents
A kind of distributed meta-data management method and system for creating strategy in advance based on file Download PDFInfo
- Publication number
- CN109344122A CN109344122A CN201811196310.XA CN201811196310A CN109344122A CN 109344122 A CN109344122 A CN 109344122A CN 201811196310 A CN201811196310 A CN 201811196310A CN 109344122 A CN109344122 A CN 109344122A
- Authority
- CN
- China
- Prior art keywords
- metadata
- file
- request
- create
- index table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Abstract
The invention discloses a kind of to create tactful distributed meta-data management method and system based on file in advance, the present invention increases proxy server in the meta data server front end of distributed file system, the file information is pre-created to record meta data server by File Index Table in local in proxy server, for intensive metadata request to create, then first distribution is pre-created the file information and returns the result to client directly from File Index Table, then Timing Synchronization is to meta data server；For metadata read-write requests, then preferentially judge whether hit file concordance list, the direct read/write File Index Table if hit.The present invention optimizes the metadata management of high performance distributed file system, storage service is provided with this to handle for large-scale data, give full play to I/O ability powerful in distributed memory system, it realizes the effective integration of high-performance and big data field, provides high performance shared storage service for various large-scale data processing platforms and application.
Description
Technical field
The invention belongs to the field of filesystems of mass data storage, and in particular to one kind is in distributed file system
The method and system of metadata are managed by the way that the strategy of file is pre-created.
Background technique
Due to the rapid development of internet, cause the transimission and storage demand of massive information increasing.Especially big number
According to the arrival in epoch, the mass small documents that internet generates need the support of High Performance Cache and Memory System, and this further improves big
Data are merged with high-performance field.The distributed file system in high-performance field can provide the storage service of fine performance, mesh
Preceding distributed file system is widely used in various big data processing platforms.Distributed file system is mainly taken by metadata
Business device and data server are constituted, and client first has to take by metadata in the file read and write in distributed file system
Device be engaged in obtain the metadata information of file, metadata information contains the location information to data, thus asking client
It asks and is forwarded on correct data server, client is enabled correctly to carry out data access.According to statistics, mass storage system (MSS)
In have at least that more than half file system operation can all be related to metadata, and existing distributed file system is in processing sea
Performance when measuring small documents read-write is very poor, can also there is asking for Single Point of Faliure for single meta data server
Topic, these obstacles of metadata seriously reduce the overall performance of distributed file system, and handle number for big data field
When according to intensive applications, a large amount of temporary file is often generated during handling due to big data, this allows for metadata
The access of server can become very frequently, to generate very big impact to system performance.Therefore, how to solve be distributed
The storage demand that formula field of filesystems handles mass small documents data is an extremely important and meaningful research.
Currently, in terms of to metadata management in distributed file system being deposited with single meta data server
The metadata in distributed file system is stored up, this mode is fairly simple for the management of metadata, and disadvantage is detrimental to system
The extension of performance；And due to only one meta data server, a large amount of metadata is read in processing distributed file system
When write request, the meta data server of this distributed file system often has Single Point of Faliure.Some researchers
Devise distributed metadata management method, by the metadata information of distributed file system with metadata server cluster come
It is managed, this method can eliminate single Problem of Failure, it is also possible to obtain good access concurrency.But due to first number
According to distributivity, increase the lookup of a metadata again when obtaining the metadata of file, increase the complexity of file access
Property and network overhead.
In traditional distributed file system: GFS, HDFS, Lustre, PVFS distributed file system are all to use
The metadata management mode of single node, there are bottlenecks for scalability aspect；Panasas distributed file system uses distributed
Metadata management strategy is not suitable for general application scenarios it require that special hardware is supported；The distributed text of Ceph
Part system realizes the strategy of distributed meta-data management by the way of the division of dynamic subtree, by the NameSpace of metadata
Be divided on different metadata management nodes by different subtrees, this mode can solve the expansion of distributed file system
Malleability problem, once but NameSpace change, will also result in a large amount of Data Migration, to cause network very big
Expense.
Although traditional distributed file system can provide high performance storage service, large-scale sea is once encountered
This high performance advantage will be difficult to embody when measuring storage demand.Although metadata the space occupied very little, metadata
Accessed frequency is unusual height, and it is current extensive that the metadata management in traditional distributed file system is difficult processing
The storage demand of data.The characteristics of for large-scale data processing, in order to which high performance distributed file system is used for big number
According to process field, some researchers are handled by filing the small documents of magnanimity to be merged into big file, though
The access pressure of meta data server so can be reduced to a certain extent, but also brings additional expense in this way, be unfavorable for point
The high performance performance of cloth file system.For large-scale data processing storage demand, if with distributed file system come
Storage service is provided for it, meta data server is the letter that must pass in face of magnanimity metadata hair access performance problem
Ditch.
Summary of the invention
In view of the above-mentioned problems existing in the prior art, in order to metadata of distributed type file system access bottleneck is effectively relieved
Problem, the technical problem to be solved by the present invention is providing a kind of distributed file system member number for pre-creating strategy based on file
According to management method and system, the present invention can optimize the metadata management of high performance distributed file system, with this
Storage service is provided to handle for large-scale data, I/O ability powerful in distributed memory system can be given full play to, is realized
The effective integration of high-performance and big data field, for various large-scale data processing platforms and application provide it is high performance shared
Storage service.
In order to solve the above-mentioned technical problem, the technical solution adopted by the present invention are as follows:
A kind of distributed meta-data management method for creating strategy in advance based on file, in the meta data server of distributed file system
Front end increases the proxy server for handling the metadata request of client, and the proxy server executes metadata management
The detailed step of management includes:
1) File Index Table that the file information is pre-created for recording meta data server is initialized；
2) metadata request of client is waited, if the metadata request for receiving client is metadata request to create, is jumped
Turn to execute step 3)；If metadata request is metadata read-write requests, execution step 6) is jumped；
3) judge whether metadata request to create is intensive metadata request to create, is then jumped if it is intensive metadata request to create
Turn to execute step 4)；Otherwise, execution step 5) is jumped；
4) by File Index Table can allocated items distribute to metadata request to create and to client return response result, and
Allocated items in File Index Table are synchronized to meta data server by timing, are exited；
5) metadata request to create is transmitted to meta data server, and executes metadata creation operation in meta data server and returns
After returning response results, the response results that meta data server returns are pass on to client, are exited；
6) whether have hit project, if there is hitting project, then directly in File Index Table if being inquired in File Index Table
Hit project carries out metadata read-write operation and to client return response result, and periodically by the hit item in File Index Table
Purpose write operation is synchronized to meta data server；If not hitting project, metadata read-write requests are transmitted to metadata
Server, and after meta data server executes metadata read-write operation return response result, metadata clothes are pass on to client
The response results that business device returns.
Preferably, each project of File Index Table includes the file record item that the file information is pre-created in step 1)
Serial number, metadata tag, filename and whether allocation identification, file record serial number is the globally unique mark of File Index Table
Know, metadata tag is the creation time and creation sequence number information for including meta data server, and filename is that text is pre-created
The default file name of part information, if allocation identification indicates that this is pre-created whether file should distribute.
Preferably, the format of the metadata tag is timestamp_n, wherein timestamp meta data server
Creation time stamp, n-tuple according to server creation sequence number information.
Preferably, judge whether metadata request to create is that intensive metadata request to create specifically refers in step 3): sentencing
The quantity of the metadata request to create received in disconnected section at the appointed time is more than whether preset threshold is true, is sentenced if setting up
Metadata request to create before settled is intensive metadata request to create；Otherwise determine that current metadata request to create is non-close
Set metadata request to create.
Preferably, the detailed step in step 4) includes:
4.1) check File Index Table in whether still have can allocated items, execution step can be jumped if allocated items if still had
4.3)；Otherwise, it jumps and executes step 4.2)；
4.2) file request is pre-created to meta data server transmission, and file is pre-created in meta data server execution and asks
Local File Index Table is updated after asking return that the file information is pre-created, so that updated File Index Table includes to create in advance
Build the file information constitute it is new can allocated items；
4.3) by File Index Table can allocated items distribute to metadata request to create and to client return response result,
And the allocated items in File Index Table are synchronized to meta data server by timing, are exited.
Preferably, step 1) initializes the File Index Table that the file information is pre-created for recording meta data server
When, the information that the file information is pre-created is that meta data server triggers in advance in client carry distributed file system
What first creation file request returned is pre-created the file information.
The present invention also provides a kind of to create tactful distributed meta-data management system, including client, member based on file in advance
Data server and data server, it is characterised in that further include increased for handling client in meta data server front end
Metadata request proxy server, and the proxy server is programmed to perform that the present invention is aforementioned to create plan based on file in advance
The step of distributed meta-data management method omited.
The present invention also provides a kind of to create tactful distributed meta-data management system based on file in advance, in distributed field system
The meta data server front end of system increases the proxy server for handling the metadata request of client, and the agency service
Device includes:
The file information is pre-created for recording meta data server for initializing in File Index Table initialization program unit
File Index Table；
Client request processing routine unit, for waiting the metadata request of client, if receiving the metadata of client
Request is metadata request to create, then jumps and execute the intensive determining program unit of request to create；If metadata request is first number
According to read-write requests, then jumps and execute read-write operation request processing routine unit；
The intensive determining program unit of request to create, for judging whether metadata request to create is intensive metadata request to create,
It is then jumped if it is intensive metadata request to create and executes intensive request to create processing routine unit；Otherwise, it is non-close to jump execution
Collect request to create processing routine unit；
Intensive request to create processing routine unit, for by File Index Table can allocated items distribute to metadata creation and ask
It asks and the allocated items in File Index Table is synchronized to meta data server to client return response result, and periodically,
It exits；
Non-dense set request to create processing routine unit, for metadata request to create to be transmitted to meta data server, and in member
After data server executes metadata creation operation return response result, the response that meta data server returns is pass on to client
As a result, exiting；
Whether read-write operation request processing routine unit has hit project for inquiring in File Index Table, if there is hit
Project, then directly carry out metadata read-write operation to the hit project in File Index Table and to client return response result,
And the hit item purpose write operation in File Index Table is synchronized to meta data server by timing；If not hitting project,
Metadata read-write requests are transmitted to meta data server, and execute metadata read-write operation returning response in meta data server
As a result after, the response results that meta data server returns are pass on to client.
Preferably, whether the intensive determining program unit judges metadata request to create of the request to create is intensive metadata
Request to create specifically refers to: judge the quantity of the metadata request to create received in section at the appointed time is more than preset threshold
No establishment determines that current metadata request to create is intensive metadata request to create if setting up；Otherwise determine currently
Metadata request to create is non-dense set metadata request to create.
Preferably, the intensive request to create processing routine unit includes:
Can allocated items check subroutine module, for check in File Index Table whether still have can allocated items, if still had
Can allocated items then jump execution can allocated items check subroutine module；Otherwise, it jumps execution and request subprogram is pre-created
Module；
Request subroutine module is pre-created, for file request to be pre-created to meta data server transmission, and in metadata
Server execution, which is pre-created after the file information is pre-created in file request return, updates local File Index Table, so that updating
File Index Table afterwards include be pre-created the file information constitute it is new can allocated items；It executes intensive request to create and handles sub- journey
Sequence module；
Intensive request to create processing subroutine module, for by File Index Table can allocated items distribute to metadata creation
It requests and the allocated items in File Index Table is synchronized to Metadata Service to client return response result, and periodically
Device exits.
Compared with prior art, the present invention has following technological merit:
1, batch documents metadata is pre-created in meta data server, can reduce client metadata access delay.
Under technical solution of the present invention, the text of batch can be pre-created in background system free time in meta data server
Part metadata record, and send these metadata informations on the proxy server in the present invention, proxy server is set up
One File Index Table manages these metadata informations.When client sends metadata request into distributed file system
When, proxy server needs the mode for searching and distributing file record item only to return to the metadata information of client needs, is not required to
Information metadata information is created again, this process eliminates opening for metadata creation compared with traditional distributed document
Pin, can be reduced the access delay of client metadata, especially when metadata request to create is more frequent, the present invention can
With the access performance of apparent promotion metadata, the metadata access delay of client is reduced.
2, the introducing of proxy server can reduce the access load of meta data server.
Under technical solution of the present invention, by introducing proxy server, proxy server in metadata management level
It coordinates with meta data server, the common management for completing metadata.It is asked when receiving the metadata access from client
It asks, which is handled by proxy server first, and proxy server goes local file concordance list to look into according to the information of request
It looks for and distributes, and then shared the access pressure of meta data server.Even if the unallocated file record item in File Index Table
Fast use is over, and proxy server can be to meta data server application wound request in advance again, and the metadata information created in advance is integrated
Into the File Index Table of proxy server, these operations are carried out on backstage, hardly influence the visit of file system
Ask performance.In addition metadata access request directly can be transmitted to Metadata Service again by the Switcher module of proxy server
The load of device, meta data server and proxy server can be also balanced.
3, high-performance is merged with big data field, can provide high performance storage service for large-scale data processing.
Under technical solution of the present invention, distributed file system can provide high performance for large-scale data processing
Storage server demand.The metadata request generated in large-scale data processing is often magnanimity, these metadata request meetings
It is directly handled by the proxy server in the present invention, when being carved with the metadata request of magnanimity when a certain and sending, agency's clothes
Business device is by simply tabling look-up and assigning process can be very good to service the request.In addition distributed file system itself is excellent
Gesture, when providing storage server for large-scale data application, data I/O is also enhanced, the height of distributed file system
Performance, shared advantage also can play very well under this large-scale data processing.
Detailed description of the invention
It in order to more clearly explain the technical solutions in the embodiments of the present application, below will be to needed in the embodiment
Attached drawing is briefly described, it should be apparent that, the accompanying drawings in the following description is only the embodiment of the present invention, general for this field
For logical technical staff, without creative efforts, other attached drawings can also be obtained according to the attached drawing of offer.
Fig. 1 is the cluster topology schematic diagram of the distributed file system of the embodiment of the present invention.
Fig. 2 is the basic procedure schematic diagram of present invention method.
Fig. 3 is distributed file system in the embodiment of the present invention to the process flow of metadata request to create.
Fig. 4 is distributed file system in the embodiment of the present invention to the process flow of metadata read-write requests.
Specific embodiment
As shown in Figure 1, the present embodiment creates the distributed meta-data management method of strategy based on file in advance, in distributed document
The meta data server front end of system increases the proxy server for handling the metadata request of client, as shown in Fig. 2, and
The detailed step that the proxy server executes metadata management management includes:
1) File Index Table that the file information is pre-created for recording meta data server is initialized；
2) metadata request of client is waited, if the metadata request for receiving client is metadata request to create, is jumped
Turn to execute step 3)；If metadata request is metadata read-write requests, execution step 6) is jumped；
3) judge whether metadata request to create is intensive metadata request to create, is then jumped if it is intensive metadata request to create
Turn to execute step 4)；Otherwise, execution step 5) is jumped；
4) by File Index Table can allocated items distribute to metadata request to create and to client return response result, and
Allocated items in File Index Table are synchronized to meta data server by timing, are exited；
5) metadata request to create is transmitted to meta data server, and executes metadata creation operation in meta data server and returns
After returning response results, the response results that meta data server returns are pass on to client, are exited；
6) whether have hit project, if there is hitting project, then directly in File Index Table if being inquired in File Index Table
Hit project carries out metadata read-write operation and to client return response result, and periodically by the hit item in File Index Table
Purpose write operation is synchronized to meta data server；If not hitting project, metadata read-write requests are transmitted to metadata
Server, and after meta data server executes metadata read-write operation return response result, metadata clothes are pass on to client
The response results that business device returns.
Traditional large-scale distributed file system consists of three parts: client (Client), meta data server
(Meta Server) and data server (Data Server).Wherein client is to meta data server and data storage service
Device sends read-write requests, and meta data server and data server request to make a response to it.The present embodiment is on the basis of above
The process flow for modifying metadata request, increases the generation for handling the metadata request of client in meta data server front end
It manages server (Proxy Server), specific framework is as shown in Figure 1.After increasing proxy server, client and data service
The function of device is identical as traditional distributed file system；For meta data server part, changing metadata management of the present invention
Process increases proxy server.There is a switching module (Switcher) inside Proxy Sever, for being responsible for according to next
It is handled from the different metadata request types of client to decide whether to be transmitted to meta data server.In Fig. 1
In, arrow 1 and 2 represents the metadata operation request and corresponding response message that client is sent to Proxy Server, arrow 3
And 4 represent data communication between client and data server, arrow 5 refers to that meta data server is sent to proxy server
The information that pre-creates of file；Dotted arrow 6 and 7 indicates the data simultaneously operating on backstage, and dotted arrow 6 indicates File Index Table
Relevant information, dotted arrow 7 indicate the heartbeat message from data server.
Proxy Server is directly interacted with client, to respond the metadata request from client；As shown in figure 1
Arrow 1 shown in.Metadata request from client is exactly the creation of metadata and the read-write of metadata nothing but, for this two
The process flows of these two types of requests are set forth in detail behind the present invention for the metadata request of seed type.Have in Proxy Server
One Switcher module, the major function of this module be according to request the characteristics of come determine metadata request whether
Directly processing still needs that meta data server is transferred to be handled on Proxy Server；Proxy Server preserves one
File Index Table, this File Index Table record entire distributed file system and are used to provide for intensive metadata request
The file metadata information that service is pre-created and opens.For as this kind of application of Hadoop, in the process of MapReduce
It is middle can the instant a large amount of temporary file of generation, at this moment Proxy Server can be using File Index Table come for this kind of member number
Efficient storage service is provided according to request.The information meeting timing of this File Index Table passes through the arrow 6 in Fig. 1 by background process
Process be synchronized on meta data server, to guarantee the metadata consistency of entire distributed file system.Work as Proxy
When File Index Table in Server is not enough, a large amount of metadata information can be also pre-created in meta data server again, and
These metadata items are transferred to Proxy Server, thus to provide instant, efficient sea for large-scale data processing
Measure storage performance.
In the present embodiment, each project of File Index Table includes that the file note of the file information is pre-created in step 1)
Record item serial number, metadata tag (metadata Tag), filename and whether allocation identification, file record serial number is file index
The globally unique identifier of table, metadata tag are the creation time and creation sequence number information for including meta data server, file
Name is the default file name that the file information is pre-created, if allocation identification indicates that this is pre-created whether file should distribute.One
As for, the settable threshold value of the number of entry of File Index Table, if the number of entry of File Index Table is more than default threshold
Value, then step 4) timing, then will be a certain number of by File Index Table after allocated items are synchronized to meta data server
Allocated items are deleted from File Index Table, so that the number of entry of File Index Table is no more than threshold value.
In the present embodiment, the format of the metadata tag is timestamp_n, wherein timestamp Metadata Service
The creation time of device is stabbed, n-tuple according to server creation sequence number information.
In the present embodiment, step 1) initializes the file index that the file information is pre-created for recording meta data server
When table, the information that the file information is pre-created is meta data server triggering in client carry distributed file system
File request return is pre-created is pre-created the file information.
Judge whether metadata request to create is that intensive metadata request to create is specifically in the present embodiment, in step 3)
Refer to: judging whether the quantity of the metadata request to create received in section at the appointed time is true more than preset threshold, if at
It is vertical then determine current metadata request to create be intensive metadata request to create；Otherwise determine current metadata request to create
For non-dense set metadata request to create.
In the present embodiment, the detailed step in step 4) includes:
4.1) check File Index Table in whether still have can allocated items, execution step can be jumped if allocated items if still had
4.3)；Otherwise, it jumps and executes step 4.2)；
4.2) file request is pre-created to meta data server transmission, and file is pre-created in meta data server execution and asks
Local File Index Table is updated after asking return that the file information is pre-created, so that updated File Index Table includes to create in advance
Build the file information constitute it is new can allocated items；
4.3) by File Index Table can allocated items distribute to metadata request to create and to client return response result,
And the allocated items in File Index Table are synchronized to meta data server by timing, are exited.
For the data portion in the process flow of request of data and traditional distributed file system in the present embodiment
Process flow is consistent, and for metadata process flow, the present invention devises a kind of metadata pipe towards novel metadata cluster
Reason method.For the treatment process of metadata, when client sends metadata to distributed file system designed by the present invention
When processing request, metadata processing request can be sent to Proxy Server first, there is a file on Proxy Server
Concordance list, the structure of this File Index Table are similar to the structure of traditional Relational DataBase table, every a line of File Index Table
A referred to as file record item；The composition of each file record item are as follows: file record item serial number, metadata Tag, filename are
No allocation identification.File record serial number is the unique identification of File Index Table, is considered as the major key information of this table, this value
Proxy server returns to the filec descriptor of the application of client when being client creation file；Metadata Tag is by metadata
Creation time and creation set of serial numbers on Metadata Server at, ensure that the global uniqueness of metadata item,
The presentation format of Tag are as follows: timestamp_n；This is empty for unappropriated file record item to filename, once this
A file record item is assigned away, this just records the name of corresponding file；Whether allocation identification indicates this record
Whether item is assigned.Proxy Server mainly needs to handle metadata request of two classes from client, and one kind is metadata
Request to create, another kind of is metadata read-write requests.
1. the metadata request from client is creation operation.
As shown in figure 3, distributed file system includes: to the process flow of metadata request to create
A1, when distributed file system of the client designed by carry, N number of file can be pre-created in meta data server
Metadata information, the corresponding metadata Tag of every metadata information, this metadata Tag is by creation time and creation sequence
It number is formed, format are as follows: timestamp_n；To ensure that the global uniqueness of this metadata information.Metadata Service
This N item record information is transferred to Proxy Server after the completion by the process that pre-creates of device, and meta data server then updates this
The pre- of ground sets a record, and creation sequence number is set 0；
A2, Proxy Server are received sets up local file index after the metadata information that meta data server is created in advance
Table.The structure of the table is similar to database table, including file record item serial number, metadata Tag, filename, if allocation identification.
A3, at this moment distributed file system are responsive to the metadata request from client.The metadata request of client
It is sent on Proxy Server, the Switcher module of Proxy Server judges what next to do according to request feature
Kind processing.If metadata request is intensive, Proxy Sever oneself handles the request, i.e. execution step A4；Otherwise, pass through
Switcher component by metadata request be transmitted in meta data server, i.e., execution step A6；
A4, the metadata request intensive from client is received.One is searched in the File Index Table of Proxy Server not divide
The entry is labeled as having distributed by the file record item matched；
A5, Proxy Server mark the entry in step A4 and its corresponding metadata information returns to client；Knot
Beam.
A6, it is handled by step A3, receives the metadata request from client non-dense set, Proxy Server's
Switcher module by the metadata request from client be transmitted in meta data server；
A7, file metadata is created on meta data server according to the metadata request forwarded in step A6；
The result of processing request is returned to Proxy Sever server by A8, meta data server；
A9, Proxy Server return to the processing result of different requests to client, distributed file system document creation process
Terminate；The part of request of data is then consistent with the process flow of former distributed file system；
File Index Table on Proxy Server has recorded the file record item and the also unallocated text gone out of the allocated mistake
Part entry information.Have a Switcher module inside Proxy Server, when have metadata request to create arrive at Proxy
When Server, Proxy Server passes through Switcher module first and judges whether the request to create of metadata is intensive, if not close
Collection transfers to Metadata Server then to respond the request to create of metadata, otherwise just responds metadata in Proxy Server
Request to create.Switcher module can be defined by user oneself, such as all metadata requests are directly transferred to first number
Handled or transferred to Proxy server to be handled according to server.Proxy Server inquires one in File Index Table
Not used file record item, if file record item not can be used, Proxy Server is sent out to Metadata Server
It send file to create request in advance, applies for a certain number of pre- wound file numbers again；Obviously apply for that the file number created in advance is bigger,
Proxy Server more can provide efficiently storage service, also be configured with higher requirement to Proxy Server in this way.
Metadata request to create from client can contain documentary name information, when in the File Index Table in Proxy Server
When having found suitable file record item can be used for distributing, the file names portion data of this file record item are filled with this article
The name of part, then Proxy Server is then returned to entry serial number by the entry labeled as having distributed as file
Filec descriptor after creation.If the metadata request to create from client is not intensive enough, first number can be directly transferred to
Carry out the metadata request to create at customer in response end according to server.Since the metadata information in file record item is all pre-created
, so meta data server will not generate very big pressure, eliminate for single-point present in single meta data server
Failure problems；When handling large-scale data, this method that file is pre-created can reduce the delay of document creation again.
2. the metadata request from client is read-write operation.
As shown in figure 4, distributed file system includes: to the process flow of metadata request to create
The interface that the distributed file system of B1, application program through the invention provides issues the read-write requests to metadata, should
Request is sent directly to Proxy Server, and Proxy Server responds its request；
B2, Proxy Server receive the metadata read-write requests from client, are believed using the filename in client request
Breath inquires corresponding entry from the File Index Table of Proxy Server；
Query result during B3, judgment step B 2, if inquire it is corresponding as a result, if its yuan updated according to request type
Data record item then immediately proceeds to step B5；If do not inquire it is corresponding as a result, if metadata read-write requests are transmitted to this
Meta data server in invention, it may be assumed that enter step B4；
B4, the metadata read-write requests from client are handled on meta data server；It is arrived according to the solicited message of client
Metadata information is obtained on meta data server, and metadata information is updated according to request type；
B5, return client metadata read-write requests processing result；
The read-write operation of metadata is very extensive in distributed file system.General distributed file system can be in client
End caches some metadata informations, first can be in the metadata cache of client local when application is written and read file
It takes and searches corresponding metadata information, then will not send metadata read-write requests to meta data server when searching.When
When Proxy Server receives the metadata read-write requests from client, using the file name information in the request come pair
Metadata is indexed.Proxy Server can go in File Index Table to inquire corresponding file note according to the filename in request
Item is recorded, due to that only need to have been marked as being searched in allocated file record item, so saving the traversal of entry
Expense.If finding corresponding entry, in the corresponding metadata information of ProxyServer local update；If otherwise request
Metadata information not in Proxy Server, then Proxy Server can forward the request to meta data server, continue
The corresponding metadata information of the request is obtained from meta data server, if not finding, illustrates the corresponding first number of the request
It is believed that breath is not present, if finding corresponding record, corresponding metadata information need to be only changed on meta data server.
The metadata request that client is sent to distributed file system can divide two types according to the characteristics of request: metadata operation
Intensive and non-dense set type.By the switcher component in Proxy Sever server come according to both classes in the present invention
The request of type comes whether decision is forwarded metadata request.The manager of metadata is essentially consisted in view of improvement of the invention
Face remains the data management in distributed file system original process flow.
The present invention also provides a kind of to create tactful distributed meta-data management system, including client, member based on file in advance
Data server and data server, it is characterised in that further include increased for handling client in meta data server front end
Metadata request proxy server, and the proxy server is programmed to perform that the present invention is aforementioned to create plan based on file in advance
The step of distributed meta-data management method omited.
The present invention also provides a kind of to create tactful distributed meta-data management system based on file in advance, in distributed field system
The meta data server front end of system increases the proxy server for handling the metadata request of client, and the agency service
Device includes:
The file information is pre-created for recording meta data server for initializing in File Index Table initialization program unit
File Index Table；
Client request processing routine unit, for waiting the metadata request of client, if receiving the metadata of client
Request is metadata request to create, then jumps and execute the intensive determining program unit of request to create；If metadata request is first number
According to read-write requests, then jumps and execute read-write operation request processing routine unit；
The intensive determining program unit of request to create, for judging whether metadata request to create is intensive metadata request to create,
It is then jumped if it is intensive metadata request to create and executes intensive request to create processing routine unit；Otherwise, it is non-close to jump execution
Collect request to create processing routine unit；
Intensive request to create processing routine unit, for by File Index Table can allocated items distribute to metadata creation and ask
It asks and the allocated items in File Index Table is synchronized to meta data server to client return response result, and periodically,
It exits；
Non-dense set request to create processing routine unit, for metadata request to create to be transmitted to meta data server, and in member
After data server executes metadata creation operation return response result, the response that meta data server returns is pass on to client
As a result, exiting；
Whether read-write operation request processing routine unit has hit project for inquiring in File Index Table, if there is hit
Project, then directly carry out metadata read-write operation to the hit project in File Index Table and to client return response result,
And the hit item purpose write operation in File Index Table is synchronized to meta data server by timing；If not hitting project,
Metadata read-write requests are transmitted to meta data server, and execute metadata read-write operation returning response in meta data server
As a result after, the response results that meta data server returns are pass on to client.
Preferably, whether the intensive determining program unit judges metadata request to create of the request to create is intensive metadata
Request to create specifically refers to: judge the quantity of the metadata request to create received in section at the appointed time is more than preset threshold
No establishment determines that current metadata request to create is intensive metadata request to create if setting up；Otherwise determine currently
Metadata request to create is non-dense set metadata request to create.
Preferably, the intensive request to create processing routine unit includes:
Can allocated items check subroutine module, for check in File Index Table whether still have can allocated items, if still had
Can allocated items then jump execution can allocated items check subroutine module；Otherwise, it jumps execution and request subprogram is pre-created
Module；
Request subroutine module is pre-created, for file request to be pre-created to meta data server transmission, and in metadata
Server execution, which is pre-created after the file information is pre-created in file request return, updates local File Index Table, so that updating
File Index Table afterwards include be pre-created the file information constitute it is new can allocated items；It executes intensive request to create and handles sub- journey
Sequence module；
Intensive request to create processing subroutine module, for by File Index Table can allocated items distribute to metadata creation
It requests and the allocated items in File Index Table is synchronized to Metadata Service to client return response result, and periodically
Device exits.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of distributed meta-data management method for creating strategy in advance based on file, which is characterized in that in distributed file system
Meta data server front end increase the proxy server for handling the metadata request of client, and the proxy server
Execute metadata management management detailed step include:
1) File Index Table that the file information is pre-created for recording meta data server is initialized；
2) metadata request of client is waited, if the metadata request for receiving client is metadata request to create, is jumped
Turn to execute step 3)；If metadata request is metadata read-write requests, execution step 6) is jumped；
3) judge whether metadata request to create is intensive metadata request to create, is then jumped if it is intensive metadata request to create
Turn to execute step 4)；Otherwise, execution step 5) is jumped；
4) by File Index Table can allocated items distribute to metadata request to create and to client return response result, and
Allocated items in File Index Table are synchronized to meta data server by timing, are exited；
5) metadata request to create is transmitted to meta data server, and executes metadata creation operation in meta data server and returns
After returning response results, the response results that meta data server returns are pass on to client, are exited；
6) whether have hit project, if there is hitting project, then directly in File Index Table if being inquired in File Index Table
Hit project carries out metadata read-write operation and to client return response result, and periodically by the hit item in File Index Table
Purpose write operation is synchronized to meta data server；If not hitting project, metadata read-write requests are transmitted to metadata
Server, and after meta data server executes metadata read-write operation return response result, metadata clothes are pass on to client
The response results that business device returns.
2. the distributed meta-data management method according to claim 1 for creating strategy in advance based on file, which is characterized in that step
It is rapid 1) in File Index Table each project include the file record item serial number that the file information is pre-created, metadata tag,
Filename and whether allocation identification, file record serial number is the globally unique identifier of File Index Table, and metadata tag is packet
The creation time and creation sequence number information of meta data server are included, filename is the default file that the file information is pre-created
Name, if allocation identification indicates that this is pre-created whether file should distribute.
3. the distributed meta-data management method according to claim 2 for creating strategy in advance based on file, which is characterized in that institute
The format for stating metadata tag is timestamp_n, wherein the creation time stamp of timestamp meta data server, n-tuple evidence
The creation sequence number information of server.
4. the distributed meta-data management method according to claim 1 for creating strategy in advance based on file, which is characterized in that step
It is rapid 3) in judge whether metadata request to create is that intensive metadata request to create specifically refers to: judge at the appointed time in section
The quantity of the metadata request to create received is more than whether preset threshold is true, and current metadata creation is determined if setting up
Request is intensive metadata request to create；Otherwise determine current metadata request to create for non-dense set metadata request to create.
5. the distributed meta-data management method according to claim 1 for creating strategy in advance based on file, which is characterized in that step
It is rapid 4) in detailed step include:
4.1) check File Index Table in whether still have can allocated items, execution step can be jumped if allocated items if still had
4.3)；Otherwise, it jumps and executes step 4.2)；
4.2) file request is pre-created to meta data server transmission, and file is pre-created in meta data server execution and asks
Local File Index Table is updated after asking return that the file information is pre-created, so that updated File Index Table includes to create in advance
Build the file information constitute it is new can allocated items；
4.3) by metadata request to create creation file filename alternate file concordance list in can allocated items text
Part name to by File Index Table can allocated items distribute to metadata request to create, to client return response result,
And the allocated items in File Index Table are synchronized to meta data server by timing, are exited.
6. the distributed meta-data management method according to claim 1 for creating strategy in advance based on file, which is characterized in that step
Rapid 1) initialization is described that file is pre-created when the File Index Table of the file information is pre-created for recording meta data server
The information of information, which triggers for meta data server in client carry distributed file system, is pre-created file request return
The file information is pre-created.
7. a kind of distributed meta-data management system for creating strategy in advance based on file, including client, meta data server sum number
According to server, it is characterised in that further include increased for handling the metadata request of client in meta data server front end
Proxy server, and the proxy server is programmed to perform described in any one of claim 1~6 and is created in advance based on file
The step of distributed meta-data management method of strategy.
8. a kind of distributed meta-data management system for creating strategy in advance based on file, which is characterized in that in distributed file system
Meta data server front end increase the proxy server for handling the metadata request of client, and the proxy server
Include:
The file information is pre-created for recording meta data server for initializing in File Index Table initialization program unit
File Index Table；
Client request processing routine unit, for waiting the metadata request of client, if receiving the metadata of client
Request is metadata request to create, then jumps and execute the intensive determining program unit of request to create；If metadata request is first number
According to read-write requests, then jumps and execute read-write operation request processing routine unit；
The intensive determining program unit of request to create, for judging whether metadata request to create is intensive metadata request to create,
It is then jumped if it is intensive metadata request to create and executes intensive request to create processing routine unit；Otherwise, it is non-close to jump execution
Collect request to create processing routine unit；
Intensive request to create processing routine unit, for by File Index Table can allocated items distribute to metadata creation and ask
It asks and the allocated items in File Index Table is synchronized to meta data server to client return response result, and periodically,
It exits；
Non-dense set request to create processing routine unit, for metadata request to create to be transmitted to meta data server, and in member
After data server executes metadata creation operation return response result, the response that meta data server returns is pass on to client
As a result, exiting；
Whether read-write operation request processing routine unit has hit project for inquiring in File Index Table, if there is hit
Project, then directly carry out metadata read-write operation to the hit project in File Index Table and to client return response result,
And the hit item purpose write operation in File Index Table is synchronized to meta data server by timing；If not hitting project,
Metadata read-write requests are transmitted to meta data server, and execute metadata read-write operation returning response in meta data server
As a result after, the response results that meta data server returns are pass on to client.
9. the distributed meta-data management system according to claim 8 for creating strategy in advance based on file, which is characterized in that institute
State whether the intensive determining program unit judges metadata request to create of request to create is that intensive metadata request to create specifically refers to:
The quantity for judging the metadata request to create received in section at the appointed time is more than whether preset threshold is true, if setting up
Determine that current metadata request to create is intensive metadata request to create；Otherwise determine that current metadata request to create is non-
Intensive metadata request to create.
10. the distributed meta-data management system according to claim 8 for creating strategy in advance based on file, which is characterized in that
The intensive request to create processing routine unit includes:
Can allocated items check subroutine module, for check in File Index Table whether still have can allocated items, if still had
Can allocated items then jump execution can allocated items check subroutine module；Otherwise, it jumps execution and request subprogram is pre-created
Module；
Request subroutine module is pre-created, for file request to be pre-created to meta data server transmission, and in metadata
Server execution, which is pre-created after the file information is pre-created in file request return, updates local File Index Table, so that updating
File Index Table afterwards include be pre-created the file information constitute it is new can allocated items；It executes intensive request to create and handles sub- journey
Sequence module；
Intensive request to create processing subroutine module, for by File Index Table can allocated items distribute to metadata creation
It requests and the allocated items in File Index Table is synchronized to Metadata Service to client return response result, and periodically
Device exits.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811196310.XA CN109344122B (en) | 2018-10-15 | 2018-10-15 | Distributed metadata management method and system based on file pre-creation strategy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811196310.XA CN109344122B (en) | 2018-10-15 | 2018-10-15 | Distributed metadata management method and system based on file pre-creation strategy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109344122A true CN109344122A (en) | 2019-02-15 |
| CN109344122B CN109344122B (en) | 2020-05-15 |
Family
ID=65310110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811196310.XA Active CN109344122B (en) | 2018-10-15 | 2018-10-15 | Distributed metadata management method and system based on file pre-creation strategy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344122B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885552A (en) * | 2019-02-18 | 2019-06-14 | 天固信息安全系统(深圳)有限责任公司 | The metadata dynamic management approach and distributed file system of distributed file system |
| CN111600949A (en) * | 2020-05-14 | 2020-08-28 | 上海鸿翼软件技术股份有限公司 | Data transmission method, device, equipment and computer readable storage medium |
| CN113111034A (en) * | 2021-04-07 | 2021-07-13 | 山东英信计算机技术有限公司 | Index pre-allocation method and device |
| CN113285992A (en) * | 2021-05-14 | 2021-08-20 | 中国邮政储蓄银行股份有限公司 | Registration center system and micro-service system |
| WO2021189308A1 (en) * | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Delete operation in object storage system using enhanced meta structure |
| CN116010348A (en) * | 2023-01-31 | 2023-04-25 | 北京基调网络股份有限公司 | Distributed mass object management method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040068579A1 (en) * | 2002-08-13 | 2004-04-08 | International Business Machines Corporation | System and method to refresh proxy cache server objects |
| CN1625109A (en) * | 2003-12-01 | 2005-06-08 | 国际商业机器公司 | Method and apparatus for virtualizing network resources |
| CN101079902A (en) * | 2007-06-29 | 2007-11-28 | 清华大学 | A great magnitude of data hierarchical storage method |
| CN101916289A (en) * | 2010-08-20 | 2010-12-15 | 浙江大学 | Method for establishing digital library storage system supporting mass small files and dynamic backup number |
| CN102523285A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Storage caching method of object-based distributed file system |
| TW201630393A (en) * | 2015-02-06 | 2016-08-16 | Plustek Inc | Method to execute cross-server document synchronization through external proxy |
| CN105868333A (en) * | 2016-03-28 | 2016-08-17 | 金蝶软件（中国）有限公司 | File processing method and device |
| CN106484821A (en) * | 2016-09-27 | 2017-03-08 | 浪潮软件集团有限公司 | Hybrid cloud storage method under cloud computing architecture |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN107832423A (en) * | 2017-11-13 | 2018-03-23 | 中山大学 | A kind of file read/write method for distributed file system |
-
2018
- 2018-10-15 CN CN201811196310.XA patent/CN109344122B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040068579A1 (en) * | 2002-08-13 | 2004-04-08 | International Business Machines Corporation | System and method to refresh proxy cache server objects |
| CN1625109A (en) * | 2003-12-01 | 2005-06-08 | 国际商业机器公司 | Method and apparatus for virtualizing network resources |
| CN101079902A (en) * | 2007-06-29 | 2007-11-28 | 清华大学 | A great magnitude of data hierarchical storage method |
| CN101916289A (en) * | 2010-08-20 | 2010-12-15 | 浙江大学 | Method for establishing digital library storage system supporting mass small files and dynamic backup number |
| CN102523285A (en) * | 2011-12-15 | 2012-06-27 | 杭州电子科技大学 | Storage caching method of object-based distributed file system |
| TW201630393A (en) * | 2015-02-06 | 2016-08-16 | Plustek Inc | Method to execute cross-server document synchronization through external proxy |
| CN105868333A (en) * | 2016-03-28 | 2016-08-17 | 金蝶软件（中国）有限公司 | File processing method and device |
| CN106484821A (en) * | 2016-09-27 | 2017-03-08 | 浪潮软件集团有限公司 | Hybrid cloud storage method under cloud computing architecture |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN107832423A (en) * | 2017-11-13 | 2018-03-23 | 中山大学 | A kind of file read/write method for distributed file system |
Non-Patent Citations (1)
| Title |
|---|
| 易建亮 等: "基于代理的并行文件系统元数据优化与实现", 《计算机研究与发展》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885552A (en) * | 2019-02-18 | 2019-06-14 | 天固信息安全系统(深圳)有限责任公司 | The metadata dynamic management approach and distributed file system of distributed file system |
| CN109885552B (en) * | 2019-02-18 | 2023-08-18 | 天固信息安全系统(深圳)有限责任公司 | Metadata dynamic management method of distributed file system and distributed file system |
| WO2021189308A1 (en) * | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Delete operation in object storage system using enhanced meta structure |
| CN111600949A (en) * | 2020-05-14 | 2020-08-28 | 上海鸿翼软件技术股份有限公司 | Data transmission method, device, equipment and computer readable storage medium |
| CN111600949B (en) * | 2020-05-14 | 2024-03-15 | 上海鸿翼软件技术股份有限公司 | Data transmission method, device, equipment and computer readable storage medium |
| CN113111034A (en) * | 2021-04-07 | 2021-07-13 | 山东英信计算机技术有限公司 | Index pre-allocation method and device |
| CN113111034B (en) * | 2021-04-07 | 2023-08-04 | 山东英信计算机技术有限公司 | Index pre-allocation method and device |
| CN113285992A (en) * | 2021-05-14 | 2021-08-20 | 中国邮政储蓄银行股份有限公司 | Registration center system and micro-service system |
| CN116010348A (en) * | 2023-01-31 | 2023-04-25 | 北京基调网络股份有限公司 | Distributed mass object management method and device |
| CN116010348B (en) * | 2023-01-31 | 2023-10-03 | 北京基调网络股份有限公司 | Distributed mass object management method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109344122B (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344122A (en) | A kind of distributed meta-data management method and system for creating strategy in advance based on file | |
| US7010617B2 (en) | Cluster configuration repository | |
| US7035931B1 (en) | Volume location service for a distributed file system | |
| EP2169909B1 (en) | System and method to maintain coherence of cache contents in a multi-tier software system aimed at interfacing large databases | |
| EP0398494B1 (en) | Maintenance of file attributes in a distributed data processing system | |
| CN111386522B (en) | System and method for data storage | |
| JP5090450B2 (en) | Method, program, and computer-readable medium for updating replicated data stored in a plurality of nodes organized in a hierarchy and linked via a network | |
| US8504523B2 (en) | Database management system | |
| US6820085B2 (en) | Web system having clustered application servers and clustered databases | |
| US6763347B1 (en) | Indexing management for hierarchical main memory | |
| CN109670089A (en) | Knowledge mapping system and its figure server | |
| US20130110873A1 (en) | Method and system for data storage and management | |
| US20100293332A1 (en) | Cache enumeration and indexing | |
| JPS63201743A (en) | Cashing for data processing system network | |
| US20100325363A1 (en) | Hierarchical object caching based on object version | |
| CN106919654A (en) | A kind of implementation method of the High Availabitity MySQL database based on Nginx | |
| US7281014B2 (en) | Method and apparatus for moving data between storage devices | |
| CN110287150A (en) | A kind of large-scale storage systems meta-data distribution formula management method and system | |
| JP3842319B2 (en) | Information retrieval system | |
| US7752225B2 (en) | Replication and mapping mechanism for recreating memory durations | |
| CN108769166A (en) | A kind of CDN cache contents managing devices based on metadata | |
| CN105138581B (en) | A kind of write-in of mobile message and read method and system | |
| CN106210038B (en) | The processing method and system of data operation request | |
| Yeo et al. | A taxonomy of issues in name systems design and implementation | |
| JP2885144B2 (en) | Management information base system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221024 Address after: 510275 No. 135 West Xingang Road, Guangzhou, Guangdong, Haizhuqu District Patentee after: SUN YAT-SEN University Patentee after: National University of Defense Technology Address before: 510275 No. 135 West Xingang Road, Guangzhou, Guangdong, Haizhuqu District Patentee before: SUN YAT-SEN University |
|
| TR01 | Transfer of patent right |