CN104123359B - Resource management method of distributed object storage system - Google Patents
Resource management method of distributed object storage system Download PDFInfo
- Publication number
- CN104123359B CN104123359B CN201410343218.7A CN201410343218A CN104123359B CN 104123359 B CN104123359 B CN 104123359B CN 201410343218 A CN201410343218 A CN 201410343218A CN 104123359 B CN104123359 B CN 104123359B
- Authority
- CN
- China
- Prior art keywords
- metadata
- store path
- server
- spid
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention discloses a resource management method of a distributed object storage system. The resource management method comprises the following steps that the distributed object storage system enables the storage path attributes of files and the metadata of the files to be separated, the storage path attributes comprise the names of storage paths where the files are placed and corresponding route access control attributes, and the metadata of the files comprise access attributes including file names, establishing times, modifying times and file lengths; the distributed object storage system comprises a storage path index server, a metadata server and object storage devices, wherein the storage path index server is used for establishing index entries for all storage paths in the system; each index entry comprises an SPID and a storage path attribute, wherein the SPID is a globally-unique storage path ID distributed for the storage paths by the storage path index server; the metadata server positions the storage path ID to the object storage device corresponding to the storage path of the SPID.
Description
Technical field
A kind of the present invention relates to field of cloud computing computer data distributed objects storage, more particularly to cloud computing environment
The method for managing resource of lower distributed objects storage system.
Background technology
Under cloud computing environment, the storage of data is an extremely important problem.Legacy network storage is broadly divided into three
Kind：Directly additional storage (Direct Attached Storage, DAS), storage area network (Storage Area Network,
) and network attached storage (Network Attached Storage, NAS) SAN.But, DAS is not suitable for large-scale data and deposits
Storage；And for SAN and NAS, with the increase of data scale, meta data server readily becomes the bottleneck of whole system.
Relative to heritage storage system, new distributed objects storage system (Object-Based Storage, OBS)
Can solve the above problems well.OBS is there is provided a traditional file-sharing with NAS system and the list of the characteristics of management
System image (single-system-image) file system, and improve the resource consolidation of SAN and extendible performance.With
The block-oriented logical block of tradition is different, and object storage does not provide the access to incoherent data block, and it is allowed by storage
Object come access correlation data block.Storage object is a pseudo-entity, and it is that a client thinks one group of related data,
Similar to the bit stream file that the size in flat file system is not limited, but object is different from file again, and it simply has attribute
Bit sequence.Although object is both different and data block, also different from file, file can be realized by the definition of Object Semanteme
Or the input/output of data block, therefore, the unification of file input/output and data block input/output by object implementatio8.
Management point two parts of object storage：Server and object storage destination end (Object-based Storage
Target, OST).The memory space of all storage destination ends is virtually unified memory space by server, while being responsible for user
NameSpace management, after certification user's request, distribute to its unique object ID, and realize user by certain algorithm
Mapping of the NameSpace to memory space.OST is responsible for the management of coupled storage device, and it does not have object naming space,
Only planar object ID space, OST are distributed memory space in storage device according to object ID or access storage object.
Safety of the object storage by credit assurance all operations, credit include allowing the operation set and integrity of client to compile
Code.To each offer credit simple to operate, even if credit is not encrypted, protection (due to Integrity Code) can be also provided, because not
A reliable credit may be provided for an operation accidentally.In order to provide safety, to some form of encipherment protection of credit it is
It is necessary.Object storage provides safety and protection to be in object level rather than entirely rolls up level, therefore allows incredible client position
In storage network, and share and access data storage is allowed without accessing the data of whole volume.As client directly can not visit
Ask positioning metadata, it is possible to provide extra protection, because can not possibly wrong configuration and anthelmintic main frame destruction positioning metadata.
Additionally, object storage architecture both provides safety in each rank：Storage system is to storage device
Certification；Certification of the storage system to calculate node；Certification of the storage system to calculate node order；All order all Jing CRC
The integrity checking of (Cyclic Redundancy Check, CRC)；Data and order are via the privately owned of IP
Property.Such level of security to user with confidence can they can efficiently be managed and easily be accessed network with more economical.Cause
This, distributed objects storage is a kind of high performance, safe mass data storage technology.
Object storage system is had been achieved for very in terms of mass data storage, performance, data sharing and extensibility
Big achievement, and obtained being widely recognized as industry.If Panasas companies are global first hand OSD (Object-based
Storage Device, object storage device) product provider, while provide it is OO storage and parallel file system
System.HP with open source code producer Lustre file system cooperations, using OSD as the expansible texts of HP StorageWork
The pith of part system.
In distributed objects storage system, all operations of data object are required for accessing metadata, extreme at some
In the case of, system occupies most data access time to the access times of metadata, consequently, it is possible to becoming system for restricting
The bottleneck of performance.So, efficient metadata management method provides high-performance and high scalability to pass to whole storage system
It is important.Meanwhile, the memory capacity of large-scale distributed storage system is often dynamic change, needs to be continuously increased new storage
Node, eliminates old memory node, and it is adaptive, balanced that high scalability, high manageability require that data object can be realized
Distribution, to ensure distribution always load balancing of the data object in whole storage system.
The content of the invention
Goal of the invention：The technical problem to be solved is for the deficiencies in the prior art, there is provided a kind of distributed
The method for managing resource of object storage system, to realize efficient metadata organization, positioning and access control, and data respectively
The dynamic load leveling of object distribution.
In order to solve above-mentioned technical problem, the invention discloses a kind of resource management side of distributed objects storage system
Method, the distributed objects storage system is by the separated from meta-data of the store path attribute of file and file, store path attribute
The store path title being located including file and corresponding path access controlled attribute, the metadata of file include filename, wound
Time, modification time and file size are built in interior access attribute；
The distributed objects storage system includes that store path index server, meta data server and object storage set
It is standby, wherein, store path index server be system in all store paths set up index entry, each index entry include SPID and
Store path attribute, wherein, SPID is the globally unique storage that store path index server distributes for store path
Path ID；Object storage devices of the meta data server positioning store path ID corresponding to the store path object of SPID；
The distributed objects storage system sets up SPID on store path index server with meta data server
Mapping table, and meta data server mapping table is divided into into master metadata server mapping table and from meta data server mapping table,
Wherein master metadata server mapping table is to map the hashed value of SPID with meta data server, is then directly will from mapping table
SPID values are mapped with meta data server；
The distributed objects storage system by SPID as index entry in store path index server major key,
According to SPID and the corresponding different meta data server of the mapping table of meta data server positioning file metadata；Metadata takes
The metadata information calculation document metadata of hashed value and store path object corresponding bucket object of the business device according to filename, and
This barrel of object is determined in object storage device.
In the present invention, new files metadata is comprised the following steps：
Step 1：Store path index server obtains the store path and corresponding access-control attributes of file, and records
In index entry；
Step 1a, store path index server obtain the store path of file and corresponding path access controlled attribute；
Step 1b, store path index server check store path access-control attributes, confirm whether user has newly-built
The authority of file metadata；
If step 1c, user have a new files metadata authority, store path index server by store path and
Corresponding access-control attributes are recorded in index entry, are otherwise terminated；
Step 2：Store path index server is that store path distributes SPID, and is recorded in index entry；
Step 3：The corresponding meta data server of positioning file metadata, file metadata storage is deposited to corresponding object
On storage equipment；
Step 3a, the mapping table for setting up SPID and meta data server on store path index server, wherein master reflect
Firing table is to map the hashed value of SPID and meta data server, is then directly by SPID values and meta data server from mapping table
Mapping；
Step 3b, according to SPID and meta data server mapping table positioning master metadata server and from Metadata Service
Device；
Step 3c, meta data server positioning store path ID are set for the object storage at the store path object place of SPID
It is standby；
Step 3d, filename is converted into into integer, according to its hashed value and the metadata information of store path object
The corresponding bucket object ID of calculation document metadata, determines this barrel of object in object storage device according to bucket object ID；
Step 3e, by file metadata store in the bucket object.
In the present invention, access file metadata and comprise the steps：
Step 4：File is decomposed into into store path and filename two parts；
Step 5：Store path index server obtains the access-control attributes and its SPID of store path
Step 5a, client have searched whether cache entry in local cache；
Step 5b, if it is not, client access store path index server, if it has, then client according to delay
Credit balance determines the bucket object of storage metadata, directly accesses metadata；
Step 5c, store path index server whether there is index entry according to the retrieval of the hashed value of store path, if
Exist, then check whether user there are access rights, if it does not exist, then terminating；If had permission, the visit in index entry is returned
Controlled attribute and SPID is asked, is otherwise terminated；
Step 5d, client obtain the index entry for returning, and save it in local caching；
Step 6：Client obtains the metadata of file；
Step 6a, client have searched whether cache entry in local cache；
Step 6b, if it is not, determining master metadata server according to the hashed value of SPID, access main metadata service
Device, if it has, then client determines the bucket object of storage metadata according to cache entry, directly accesses metadata；
Whether step 6c, master metadata server have the store path object that store path ID is SPID in searching caching,
If it did not, reading the store path object from object storage device according to SPID, otherwise, master metadata server is directly true
Surely store the bucket object of metadata；
Step 6d, filename is converted into into integer, according to its hashed value and the metadata information of store path object
The bucket object ID that calculation document metadata is located, if this barrel of object is deposited from object according to bucket object ID not in local cache
This barrel of object is read in storage equipment；
Step 6e, client obtain the metadata of file from bucket object；
If step 6f, master metadata server overload, from store path index server obtain light load from unit
Data server, the file metadata inquired about from meta data server required for obtaining；
Step 6g, client are stored in the metadata item of acquisition in local caching.
In the present invention, in currently stored path during newly-built catalogue, build new store path access-control attributes include with
Lower step：
Step 7, user inquire about the access-control attributes in currently stored path, confirm whether user has the power of newly-built catalogue
Limit, if had permission, execution step 2 otherwise terminates；
Step 8, user set the access rights attribute of newly-built catalogue；
The access rights attribute of step 9, the access-control attributes according to currently stored path and newly-built catalogue, construction are new
The access-control attributes of store path.
Present invention achieves the method for managing resource of distributed objects storage system, with advantages below：
1st, efficient metadata organization, positioning and access control are realized；
2nd, realize the high-efficiency management to data resources such as objects in system.
3rd, the problems such as solving mass data storage system resource management and be difficult, it is ensured that data under cloud storage environment
Safety.
Description of the drawings
With reference to the accompanying drawings and detailed description the present invention is done and is further illustrated, the present invention's is above-mentioned
And/or otherwise advantage will become apparent.
Fig. 1 is store path object internal structure.
Fig. 2 a and Fig. 2 b is respectively the master and slave mapping tables of MDS.
Fig. 3 is the flow chart for creating new file metadata.
Fig. 4 is the flow chart for accessing file metadata.
Fig. 5 is object resource management work content.
Specific embodiment
The characteristics of present invention is stored for distributed objects under cloud computing environment, it is proposed that a kind of distributed objects storage
Method for managing resource.
(1) metadata management
File and data separate management, metadata information and the real data read-write of file system are divided by object storage system
From alleviating the metadata live load of system, improve the overall performance of system.The present invention is by store path attribute and file
Metadata further separate, store path attribute include file be located store path title and corresponding path access control
Attribute, the metadata of file include the access attributes such as filename, creation time, modification time, file size.Directory attribute is by depositing
Storage path indexing server (Storage Path Index Server, SPIS) concrete management, is all storage roads in system
A globally unique store path ID is distributed in footpath.Directory attribute in store path index server includes：
●SPID：Represent globally unique store path ID；
●ACp：Represent the access-control attributes of the store path；
●StoragePath：Represent the store path name.
When createing directory every time, SPIS records the path access controlled attribute for obtaining in current index entry, in the mesh
Record is lower when creating new catalogue, it is no longer necessary to traversal above catalogue, only need recurrence use current path access controlled attribute with
The access-control attributes of newly-built catalogue file can build new store path access-control attributes.SPID is indexed for store path
File metadata under the major key indexed in server, store path is distributed to different first numbers by the hashed value of SPID
According to server (Metadata Server, MDS).This method arbitrarily can change file road under the conditions of SPID is immovable
Directory name and access rights in footpath, it is to avoid as the modification of directory name and access rights causes the unit of file under the store path
Data update, so as to it also avoid substantial amounts of metadata migration.
Meta data server is mainly responsible for storage and manages the metadata of data file, and these metadata are included in one kind
Referred to as in the logical structure of store path object (Storage Path Object, SPO).SPO is big comprising one or more fixations
The entry of little bucket (Bucket) object, file of each Bucket object comprising fixed qty or catalogue file metadata
Entrance (Entry), each Entry then include whole metadata of a certain file.In the original position of store path object
Comprising the metadata for describing the store path object properties, the internal structure of store path object is as shown in Figure 1.
Generally hashed in different MDS according to the value of statistical indicant of file using static hash method.The present invention combine dictionary with
Hashing, sets up the mapping table of SPID and MDS on SPIS, and MDS mapping tables is divided into main MDS mapping tables and is mapped from MDS
Table, wherein main mapping table are to map the hashed value of SPID with MDS, are then directly to map SPID values with MDS from mapping table.From
The purpose of mapping table is to prevent from transshipping the extreme case for accessing.Master and slave mapping table is as shown in Figure 2 a and 2 b.
The flow process of new files metadata is as follows, and its flow chart is as shown in Figure 3：
Step one：SPIS obtains the store path and corresponding access-control attributes of file, and records in index entry；
, SPIS obtain file the corresponding access-control attributes of store path；
, SIPS inquiry store path access-control attributes, confirm whether user has the authority of new files；
If, user have a new files authority, store path and corresponding access-control attributes record are existed by SPIS
Index entry, otherwise terminates.
Step 2：SPIS is that store path distributes SPID, and is recorded in index entry；
Step 3：The corresponding MDS of positioning file metadata, file metadata is stored on corresponding OSD；
, set up the mapping table of SPID and MDS on SPIS, wherein main mapping table is to reflect the hashed value of SPID with MDS
Penetrate, be then that directly SPID values are mapped with MDS from mapping table；
, determine main MDS according to SPID and MDS mapping tables and from MDS；
, MDS positioning store path ID for SPID the OSD that is located of store path object；
, filename is converted into into integer, calculated according to its hashed value and the metadata information of store path object
The corresponding bucket object ID of file metadata, determines this barrel of object in OSD according to bucket object ID；
, the metadata of file stored in bucket object.
The flow process for accessing file metadata is as follows, and its flow chart is as shown in Figure 4：
Step one：File is decomposed into into store path and filename two parts；
Step 2：SPIS obtains the access-control attributes and its SPID of store path：
, client searched whether cache entry in local Cache；
, if it is not, client access SPIS, if it has, then client according to cache entry determine storage metadata
Bucket object, directly access metadata；
, SPIS according to the hashed value quick-searching of store path whether there is index entry, if it is present check user
Whether access rights are had, if it does not exist, then terminating；If it has, the access-control attributes and SPID in index entry are then returned, it is no
Then terminate；
If, client obtain return index entry, save it in local Cache；
Step 3：Client obtains the metadata of file；
, client searched whether cache entry in local Cache；
, if it is not, determining main MDS according to the hashed value of SPID, access main MDS, if it has, then client according to
Cache entry determines the bucket object of storage metadata, directly accesses metadata；
, main MDS whether have the store path object that store path ID is SPID in searching Cache, if it did not, according to
SPID reads the store path object from OSD, and otherwise, master metadata server directly determines the bucket object of storage metadata；
, filename is converted into into integer, calculated according to its hashed value and the metadata information of store path object
The bucket object ID that file metadata is located, if this barrel of object reads the bucket according to bucket object ID from OSD not in Cache
Object；
, client obtain the metadata of file from bucket object；
If, main MDS overloads, from SPIS obtain light load from MDS, the file inquired about from MDS required for obtaining
Metadata；
, client is stored in the metadata item of acquisition in local caching Cache.
(2) object resource management
In object storage system, used as a kind of resource, needs are reasonably managed and are operated object data, so as to carry
High object storage system performance and resource utilization.
The management of object is mainly managed including object resource, object basic operation management, object performance management etc., every tool
Body running content is as shown in Figure 5.In object resource management, scale and power of the object storage system according to memory node in system
Unit interval is divided into the different interval of length by value, and between these interval and memory nodes sets up mapping relations.Pass through
Data object is mapped to be evenly distributed on unit interval [0,1) in set of random numbers in, according to mapping determine data object
Storage location, so as to realize object data balanced storage in systems.
Embodiment
(1) metadata management
In terms of metadata management relates generally to metadata positioning and file access control attribute management two.
The basic step of metadata positioning action is when client needs to access data file, it is necessary first to accessed
SPIS, inquiry will access the access-control attributes and SPID of the store path of file.If access rights, then root is needed
Hash calculating is carried out according to SPID, the main MDS of storage file metadata is determined.In the case of normal access, by inquiring about main MDS it is
File metadata required for can obtaining, if main MDS overloads, from SPIS obtain light load from MDS, inquire about from MDS
File metadata required for obtaining.As a example by accessing the metadata of file/home/src/test/HelloWorld.Java,
Illustrate specific access process.
Step one：Following file/home/src/test/HelloWorld.java is decomposed into into store path/home/
Src/test/ and filename HelloWorld.java two parts；
Step 2：Obtain the access-control attributes and its SPID of store path/home/src/test/；
, searched whether cache entry in local Cache；
, if it is not, access SPIS；
, SPIS index entry whether there is according to the hashed value quick-searching of/home/src/test/, if it is present
Check whether user there are access rights；If it has, then returning the access-control attributes and SPID in index entry；
If, obtain return index entry, save it in local Cache；
Step 3：Obtain the metadata of file HelloWorld.java；
, searched whether cache entry in local Cache；
, if it is not, determining main MDS according to the hashed value of SPID, access main MDS；
, main MDS whether have the store path object that store path ID is SPID in searching Cache, if it did not, according to
SPID reads the store path object from OSD；
, HelloWorld.java is converted into into integer, according to its hashed value and the metadata of store path object
Information calculates the Bucket object IDs that HelloWorld.java is located, if the Bucket objects are not in Caehe, from OSD
Read the Bucket objects；
, obtain the metadata of HelloWorld.java from Bucket objects；
If, main MDS overloads, from SPIS obtain light load from MDS, the file inquired about from MDS required for obtaining
Metadata；
, the metadata item of acquisition is stored in local Cache.
The basic operation process of file access control attribute management is by the access-control attributes of file itself and place
Store path access-control attributes separate management.During the access-control attributes of construction store path, new catalogue is created every time all
The path access controlled attribute that will be obtained is needed to record in current index entry.When creating new subdirectory under the catalogue, only
Need to visit with reference to building new store path by current path access controlled attribute and the access-control attributes of newly-built catalogue
Controlled attribute is asked, and without the need for traveling through each layer catalogue above.The benefit of this method is, it is only necessary to can inquiry text by SPIS
The store path access-control attributes of part, need not travel through all catalogues in file place path, substantially increase metadata access
Efficiency.When the following is newly-built catalogue test under catalogue/home/src, the access control of store path/home/src/test is constructed
Attribute process processed is as follows：
Step one：User inquires about the access-control attributes of store path/home/src, the access control of acquisition/home/src
Attribute processed is:Rwxr-xr-x, i.e. user have the authority of newly-built catalogue；
Step 2：User set the access rights attribute of newly-built catalogue test as：rwxr-r-；
Step 3：According to access-control attributes and the access rights of newly-built catalogue test of store path/home/src
Attribute, construct store path/home/src/test access-control attributes be:rwxr-r-.
(2) object resource management
Object resource management is mainly concerned with the resource allocation of data object, data object operation, systematic function monitoring, money
The contents such as source recovery, which comprises the following steps that：
Step one：Object storage system receives the object resource application from user, and application information includes that resource requirement is big
It is little.Resource requirement size in resource bid and itself storage capacity are contrasted by object storage system, if storage resource
Abundance, then for applying for distribution object resource, otherwise produce inadequate resource alarm.
Step 2：During object storage distribution object resource, it is that data object x produces the overall situation by linear hash function H (x)
Unique ID, and be mapped to randomly generate be evenly distributed on unit interval [0,1) on set of random numbers on.
Step 3：Object storage system by unit interval [0, l) be divided into such as dried fruitss Uncrossed separate portions.Pass through
Relatively in storage system, the weight of all memory nodes, between interval and memory node sets up mapping relations, is mapped to same
The interval of individual memory node constitutes an interval domain, and the interval in interval domain is carried out according to the order that interval left margin is incremented by
Arrangement, in each interval domain, all siding-to-siding block length sums represent the weight of memory node.
Step 4：The mapping range belonging to random number and interval domain corresponding to object storage system searching data object x,
Corresponding memory node is uniquely determined by interval domain, and by data object x storages to corresponding memory node.
Step 6：In object storage system running, system is monitored to itself storage performance, and generates performance
Record data.If systematic function is less than predetermined threshold, object storage system sends alarm.
Step 5：If user submits the request of operation object resource to, object storage system is according to user's request to object
Resource carries out associative operation.For example, if user submits to the deletion action to data object x, object to store according to object x institutes
Memory node is determined in interval domain, delete data object x, and discharge corresponding storage resource from the memory node.
The invention provides a kind of method for managing resource of distributed objects storage system, implements the technical scheme
Method and approach are a lot, and the above is only the preferred embodiment of the present invention, it is noted that common for the art
For technical staff, under the premise without departing from the principles of the invention, can also make such as dried fruitss improvements and modifications, these improve and
Retouching also should be regarded as protection scope of the present invention.In the present embodiment, clearly each ingredient not can use prior art in addition real
It is existing.
Claims (2)
1. a kind of method for managing resource of distributed objects storage system, it is characterised in that
The distributed objects storage system is by the separated from meta-data of the store path attribute of file and file, store path attribute
The store path title being located including file and corresponding path access controlled attribute, the metadata of file include filename, wound
Time, modification time and file size are built in interior access attribute；
The distributed objects storage system includes store path index server, meta data server and object storage device,
Wherein, store path index server sets up index entry for all store paths in system, and each index entry includes SPID and deposits
Storage path attribute, wherein, SPID is the globally unique storage road that store path index server distributes for store path
Footpath ID；Object storage devices of the meta data server positioning store path ID corresponding to the store path object of SPID；
The distributed objects storage system sets up the mapping of SPID and meta data server on store path index server
Table, and meta data server mapping table is divided into into master metadata server mapping table and from meta data server mapping table, wherein
Master metadata server mapping table is to map the hashed value of SPID and meta data server, is then directly by SPID from mapping table
Value is mapped with meta data server；
The distributed objects storage system by SPID as index entry in store path index server major key, according to
SPID and the corresponding different meta data server of the mapping table of meta data server positioning file metadata；Meta data server
According to the corresponding bucket object of the metadata information calculation document metadata of hashed value and store path object of filename, and right
As determining this barrel of object in storage device；
New files metadata is comprised the following steps：
Step 1：Store path index server obtains the store path and corresponding access-control attributes of file, and records in rope
Draw item；
Step 1a, store path index server obtain the store path of file and corresponding path access controlled attribute；
Step 1b, store path index server check store path access-control attributes, confirm whether user has new files
The authority of metadata；
If step 1c, user have new files metadata authority, store path index server is by store path and accordingly
Access-control attributes record in index entry, otherwise terminate；
Step 2：Store path index server is that store path distributes SPID, and is recorded in index entry；
Step 3：The corresponding meta data server of positioning file metadata, file metadata storage is set to corresponding object storage
It is standby upper；
Step 3a, the mapping table for setting up SPID and meta data server on store path index server, wherein main mapping table
It is that the hashed value of SPID is mapped with meta data server, is then that directly SPID values and meta data server are reflected from mapping table
Penetrate；
Step 3b, according to SPID and meta data server mapping table positioning master metadata server and from meta data server；
The object storage device of step 3c, meta data server positioning store path ID for the store path object place of SPID；
Step 3d, filename is converted into into integer, is calculated according to its hashed value and the metadata information of store path object
The corresponding bucket object ID of file metadata, determines this barrel of object in object storage device according to bucket object ID；
Step 3e, by file metadata store in the bucket object；
Access file metadata to comprise the steps：
Step 4：File is decomposed into into store path and filename two parts；
Step 5：Store path index server obtains the access-control attributes and its SPID of store path
Step 5a, client have searched whether cache entry in local cache；
Step 5b, if it is not, client access store path index server, if it has, then client is according to cache entry
It is determined that the bucket object of storage metadata, directly accesses metadata；
Step 5c, store path index server whether there is index entry according to the retrieval of the hashed value of store path, if it does,
Then check whether user there are access rights, if it does not exist, then terminating；If had permission, the access control in index entry is returned
Attribute processed and SPID, otherwise terminate；
Step 5d, client obtain the index entry for returning, and save it in local caching；
Step 6：Client obtains the metadata of file；
Step 6a, client have searched whether cache entry in local cache；
Step 6b, if it is not, determining master metadata server according to the hashed value of SPID, access master metadata server,
If it has, then client determines the bucket object of storage metadata according to cache entry, metadata is directly accessed；
Whether step 6c, master metadata server have the store path object that store path ID is SPID in searching caching, if
No, the store path object is read from object storage device according to SPID, otherwise, master metadata server directly determines to be deposited
The bucket object of storage metadata；
Step 6d, filename is converted into into integer, is calculated according to its hashed value and the metadata information of store path object
The bucket object ID that file metadata is located, if this barrel of object is set from object storage according to bucket object ID not in local cache
This barrel of object of standby middle reading；
Step 6e, client obtain the metadata of file from bucket object；
If step 6f, master metadata server overload, from store path index server obtain light load from metadata
Server, the file metadata inquired about from meta data server required for obtaining；
Step 6g, client are stored in the metadata item of acquisition in local caching.
2. the method for managing resource of a kind of distributed objects storage system according to claim 1, it is characterised in that working as
During the newly-built catalogue of front store path, the access-control attributes for building new store path are comprised the following steps：
Step 7, user inquire about the access-control attributes in currently stored path, confirm whether user has the authority of newly-built catalogue, such as
Fruit has permission, then execution step 2, otherwise terminate；
Step 8, user set the access rights attribute of newly-built catalogue；
The access rights attribute of step 9, the access-control attributes according to currently stored path and newly-built catalogue, construction is new to be stored
The access-control attributes in path.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410343218.7A CN104123359B (en) | 2014-07-17 | 2014-07-17 | Resource management method of distributed object storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410343218.7A CN104123359B (en) | 2014-07-17 | 2014-07-17 | Resource management method of distributed object storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104123359A CN104123359A (en) | 2014-10-29 |
| CN104123359B true CN104123359B (en) | 2017-03-22 |
Family
ID=51768770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410343218.7A Active CN104123359B (en) | 2014-07-17 | 2014-07-17 | Resource management method of distributed object storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104123359B (en) |
Families Citing this family (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105900068B (en) * | 2014-11-24 | 2019-01-15 | 华为技术有限公司 | The systems, devices and methods of path management |
| CN106527960B (en) * | 2015-09-14 | 2021-04-02 | 中兴通讯股份有限公司 | Multi-storage-disk load management method and device, file system and storage network system |
| US10353926B2 (en) * | 2015-11-17 | 2019-07-16 | Microsoft Technology Licensing, Llc | Unified activity service |
| CN105631269B (en) * | 2015-12-30 | 2018-11-27 | 成都广达新网科技股份有限公司 | A kind of cache server database resource method for detecting availability and device |
| CN107241299B (en) * | 2016-03-29 | 2021-03-02 | 百度在线网络技术（北京）有限公司 | Method and device for controlling and managing authority of network disk |
| CN106202367B (en) | 2016-07-07 | 2019-03-05 | 腾讯科技（深圳）有限公司 | A kind of processing method and processing device of object information |
| CN106339415B (en) * | 2016-08-12 | 2019-08-23 | 北京奇虎科技有限公司 | Querying method, the apparatus and system of data |
| CN106446715A (en) * | 2016-10-11 | 2017-02-22 | 武汉斗鱼网络科技有限公司 | File encryption method and device |
| CN108108376B (en) * | 2016-11-25 | 2021-07-09 | 中标软件有限公司 | Linear self-adaptive file name length scheduling system and method |
| CN108427677B (en) * | 2017-02-13 | 2023-01-06 | 阿里巴巴集团控股有限公司 | Object access method and device and electronic equipment |
| CN106980540B (en) * | 2017-03-07 | 2020-07-10 | 清华大学 | Distributed multi-dimensional discrete data calculation method |
| US10540323B2 (en) | 2017-05-30 | 2020-01-21 | Western Digital Technologies, Inc. | Managing I/O operations in a storage network |
| CN107291870B (en) * | 2017-06-15 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method for reading files in distributed storage in batch |
| CN107590249A (en) * | 2017-09-18 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of balancing method of loads of distributed file system, device and equipment |
| WO2019127234A1 (en) * | 2017-12-28 | 2019-07-04 | 华为技术有限公司 | Object migration method, device, and system |
| CN110096220B (en) | 2018-01-31 | 2020-06-26 | 华为技术有限公司 | Distributed storage system, data processing method and storage node |
| CN112417036A (en) * | 2018-09-10 | 2021-02-26 | 华为技术有限公司 | Method and device for processing metadata of object in distributed storage system |
| CN109445694B (en) * | 2018-10-19 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata separation method and device for distributed storage system |
| CN109918355B (en) * | 2019-03-05 | 2020-12-15 | 杭州前云数据技术有限公司 | Virtual metadata mapping system and method for implementing NAS based on object storage service |
| CN110012087B (en) * | 2019-03-28 | 2022-02-01 | 中国工商银行股份有限公司 | Data processing system and data processing method |
| CN110046133B (en) * | 2019-04-12 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Metadata management method, device and system for storage file system |
| CN111324799B (en) * | 2020-02-05 | 2021-05-04 | 星辰天合（北京）数据科技有限公司 | Search request processing method and device |
| CN111427862B (en) * | 2020-03-19 | 2022-11-04 | 国电南瑞科技股份有限公司 | Metadata management method for distributed file system in power grid dispatching control system |
| CN111427910A (en) * | 2020-03-20 | 2020-07-17 | 北京百度网讯科技有限公司 | Data processing method and device |
| CN111447275B (en) * | 2020-03-26 | 2021-01-01 | 深圳市中盛瑞达科技有限公司 | Storage system and storage device |
| CN111857890B (en) * | 2020-07-24 | 2021-08-31 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN114237490A (en) * | 2021-11-02 | 2022-03-25 | 清华大学 | Large-scale data storage and reading method and device based on Nauru-graph |
| CN116150093B (en) * | 2023-03-04 | 2023-11-03 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN117435559B (en) * | 2023-12-20 | 2024-03-01 | 苏州元脑智能科技有限公司 | Metadata hierarchical management method and device, storage medium and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020315A (en) * | 2013-01-10 | 2013-04-03 | 中国人民解放军国防科学技术大学 | Method for storing mass of small files on basis of master-slave distributed file system |
| CN103150394A (en) * | 2013-03-25 | 2013-06-12 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
| CN103229173A (en) * | 2012-12-26 | 2013-07-31 | 华为技术有限公司 | Metadata management method and system |
| CN103812939A (en) * | 2014-02-17 | 2014-05-21 | 李漾 | Big data storage system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060036602A1 (en) * | 2004-08-13 | 2006-02-16 | Unangst Marc J | Distributed object-based storage system that stores virtualization maps in object attributes |
-
2014
- 2014-07-17 CN CN201410343218.7A patent/CN104123359B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103229173A (en) * | 2012-12-26 | 2013-07-31 | 华为技术有限公司 | Metadata management method and system |
| CN103020315A (en) * | 2013-01-10 | 2013-04-03 | 中国人民解放军国防科学技术大学 | Method for storing mass of small files on basis of master-slave distributed file system |
| CN103150394A (en) * | 2013-03-25 | 2013-06-12 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
| CN103812939A (en) * | 2014-02-17 | 2014-05-21 | 李漾 | Big data storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104123359A (en) | 2014-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104123359B (en) | Resource management method of distributed object storage system | |
| US10467188B2 (en) | In-line policy management with multi-level object handle | |
| Liao et al. | Multi-dimensional index on hadoop distributed file system | |
| US9710535B2 (en) | Object storage system with local transaction logs, a distributed namespace, and optimized support for user directories | |
| CN106874383B (en) | Decoupling distribution method of metadata of distributed file system | |
| KR102615616B1 (en) | Key value file system | |
| US9565254B2 (en) | Object location service for network-based content repository | |
| US10019459B1 (en) | Distributed deduplication in a distributed system of hybrid storage and compute nodes | |
| CN102158546B (en) | Cluster file system and file service method thereof | |
| US9043372B2 (en) | Metadata subsystem for a distributed object store in a network storage system | |
| US8180813B1 (en) | Content repository implemented in a network storage server system | |
| CN103812939B (en) | Big data storage system | |
| CN103067461B (en) | A kind of metadata management system of file and metadata management method | |
| US20110246491A1 (en) | Method and apparatus for tiered storage | |
| US20130311520A1 (en) | Storage-service-provision apparatus,system, service-provision method, and service-provision program | |
| CN113010486B (en) | Metadata layered caching method and device for centerless distributed file system | |
| CN104408111A (en) | Method and device for deleting duplicate data | |
| US11334523B2 (en) | Finding storage objects of a snapshot group pointing to a logical page in a logical address space of a storage system | |
| CN111045857A (en) | Method for data backup and recovery, electronic device and computer readable storage medium | |
| CN103544300B (en) | A kind of extensible storage index structure in cloud environment realize method | |
| CN104182418B (en) | Node metadata acquisition methods and device | |
| Hendricks et al. | Improving small file performance in object-based storage | |
| CN103744882A (en) | Catalogue fragment expressing method and device based on key value pair | |
| US9043295B2 (en) | Providing record-level alternate-index upgrade locking | |
| CN104537023A (en) | Storage method and device for reverse index records |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 210006 Tianyuan East Road, Jiangning District, Nanjing, Jiangsu Province, No. 1 Patentee after: Zhong Tong clothing consulting and Design Research Institute Co., Ltd. Address before: 210006 Tianyuan East Road, Jiangning District, Nanjing, Jiangsu Province, No. 1 Patentee before: Jiangsu Posts & Telecommunications Planning and Designing Institute Co., Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: The Jianye District of Nanjing City, Jiangsu province 210000 Jiangdong Street No. 58 Patentee after: CHINA INFORMATION CONSULTING & DESIGNING INSTITUTE Co.,Ltd. Address before: 210006 No. 1 East Tianyuan Road, Jiangning District, Jiangsu, Nanjing Patentee before: CHINA INFORMATION CONSULTING & DESIGNING INSTITUTE Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |