CN101183318A - Resource allocation method and system - Google Patents

Resource allocation method and system Download PDF

Info

Publication number
CN101183318A
CN101183318A CNA2007101794319A CN200710179431A CN101183318A CN 101183318 A CN101183318 A CN 101183318A CN A2007101794319 A CNA2007101794319 A CN A2007101794319A CN 200710179431 A CN200710179431 A CN 200710179431A CN 101183318 A CN101183318 A CN 101183318A
Authority
CN
China
Prior art keywords
resource
file
allocation
blk
resources
Prior art date
Application number
CNA2007101794319A
Other languages
Chinese (zh)
Other versions
CN100557571C (en
Inventor
张军伟
郭明阳
张建刚
Original Assignee
中国科学院计算技术研究所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院计算技术研究所 filed Critical 中国科学院计算技术研究所
Priority to CNB2007101794319A priority Critical patent/CN100557571C/en
Publication of CN101183318A publication Critical patent/CN101183318A/en
Application granted granted Critical
Publication of CN100557571C publication Critical patent/CN100557571C/en

Links

Abstract

The invention provides a method and system of resource allocation, comprising the following steps: firstly, applying for the resource, acquiring (blk_start, blk_len) and adding (blk_offset, blk_start, blk_len) domain to the index node of the file; secondly, acquiring the resource record from the pre-allocated resource into the file layout to implement resource allocation; finally, releasing the file layout resource and releasing the preallocation resource if preallocation resource exists. The invention has the advantages of having wide application range by adopting the lasting preallocation technology, avoiding the serious space waste when the files are of large quantity, and automatically deciding granularity of preallocation by utilizing the asynchronous recovery mechanism to recover the preallocation resource.

Description

A kind of resource allocation methods and system

Technical field

The present invention relates to share the resource allocation techniques of storage system, particularly a kind of resource allocation methods and system.

Background technology

In Streaming Media etc. and other application at big document order visit, the file continuous resource distributes and can reduce that magnetic disc head moves this very big disk operating that influences disk performance in big document order access process, thereby improves the utilization factor of disk and improve the file access performance.But under the multi-process environment, when a plurality of processes are created different file simultaneously, because influencing each other between shared and a plurality of processes of storage resources caused the continuity of each file relatively poor, thereby influenced the sequential access performance of file.In addition, if the poor continuity of file, then through after deletion, the establishment etc. at random of the file resource operation, the fragmentation degree of also having aggravated file system resource also can impact the performance of file system.Therefore, guarantee that under the multi-process environment the continuous resources allocation of file is the problem that file system need solve.

The existing file system fails to overcome the above problems preferably.File layout in the ext4 file system (layout) adopts piece section (extent) mode, all records in the file layout preallocated resource is disposable, and all is labeled as vacant.There are the following problems for this mechanism: can not be applied in file layouts such as ext2 and ext3 and not adopt in the file system of piece segmented mode; When the quantity of file more for a long time because each file is all held a part of pre-allocation resource, yet this mechanism does not provide the reclaim mechanism to this class resource, thereby causes the wasting of resources; The granularity of pre-allocation resource can not generate automatically, need be made amendment by the setting of ioctl interface or by application program by the user.

Summary of the invention

The object of the invention provides a kind of resource allocation methods and system, solves the limitation problem of prior art accommodation.

For achieving the above object, according to an aspect of the present invention, provide a kind of resource allocation methods, comprised the following steps:

1) determine the pre-allocation of resources granularity, the application pre-allocation resource obtains physical resource starting block number;

2) in file inode, increase the territory, the starting block of log file expectation logical place, physical resource number and pre-allocation of resources granularity in the territory;

3) carry out resources allocation for the file destination logical place, wherein comprise, the physical resource of described physical resource starting block number expression is mapped to this document target logic position, and this map record in file layout, upgrade above-mentioned territory content for the file destination logical place adds 1, the starting block of physical resource number add 1 and the pre-allocation of resources granularity subtract 1.

According to a further aspect in the invention, above-mentioned steps 3) comprises step 4) releasing document layout resource after,, then discharge pre-allocation resource if there is pre-allocation resource.

According to another aspect of the invention, above-mentioned steps 2) comprise file destination is inserted in the pre-allocation resource file chained list; Above-mentioned steps 4) comprises from pre-allocation resource file chained list and delete this document; Also comprise the asynchronous recovery file of step 5) pre-allocation resource.

According to another aspect of the invention, provide a kind of resource allocation system, comprised:

The resource management parts that are connected with storage resources; The control assembly that is connected with the resource management parts; Wherein control assembly has the interface of receiving target file;

Wherein said control assembly is made up of lasting predistribution parts and file continuous dispensing parts, wherein, described lasting predistribution parts are used for determining the pre-allocation of resources granularity, from described resource management parts, apply for pre-allocation resource, in file inode, increase the starting block number of log file expectation logical place, physical resource and the territory of pre-allocation of resources granularity; Described file continuous dispensing parts are used for resources allocation, wherein comprise, and the physical resource of described physical resource starting block number expression is mapped to described file destination logical place, and described map record in file layout, upgrade described territory content.

According to another aspect of the invention, above-mentioned control assembly is the asynchronous recovery part of include file pre-allocation resource also, is used to reclaim idle pre-allocation resource.

Compared with prior art, beneficial effect of the present invention is: by lasting predistribution technology, make that this method and system are applied widely, can also be applicable to that except can being suitable for the ext4 file system ext2 and ext3 etc. adopt the file system of non-segment structure.And the present invention reclaims pre-allocation resource by adopting asynchronous reclaim mechanism, avoided the space waste under the huge situation of quantity of documents serious, and can determine preallocated granularity automatically.

Description of drawings

Below, describe embodiments of the invention in conjunction with the accompanying drawings in detail, wherein:

Fig. 1 is file layout structure figure in the ext4 system.

Fig. 2 is that the present invention is used to guarantee file continuous resource distribution system configuration figure.

Embodiment

In the ext4 file system, when file carries out resources allocation, file layout has adopted the piece segmented mode, pre-allocation resource with (file_offset, blk_start, mode blk_len) records in the file layout, logo file logical place (file_offset, blk_len) corresponding physical storage locations (blk_start, blk_len); In addition, the sign that is not used is set also, whether whether identifying resource is used, be the cavity to show this part resource, but that is to say not to be assigned with was not also write.

File layout structure based on the piece section in the ext4 file system is as follows:

struct?ext4_extent{

_le32ee_block;

_lel6ee_len;

_lel6ee_start_hi;

_le32ee_start;

};

Wherein, ee_block representation file logical block number (LBN); Ee_len represents the granularity of this piece section, and the ee_len total bit is 16, represents " invalid " sign with most significant digit, shows whether this piece section was modified, and that is to say it is not the cavity; 48 pieces of remaining two territories splicing becoming number, represents physical piece number.

The file layout structure as shown in Figure 1 in the ext4 system.Position and the quantity of preallocated Resource Block section in file layout all is unfixed.Fig. 1 left side is a content in the file layout, two piece sections (0 are arranged shown in it, 1000,200) and (1001,2000,6000), represent respectively the file logical place since 0, length is 1000 logical resource, be mapped to physical block number since 200, length is 1000 physical resource, and the file logical place since 1001, length is 2000 logical resource, be mapped to physical block number since 6000, length is 2000 physical resource.Because " invalid " sign does not have set, so these two piece sections are not empty resources, promptly not pre-allocation resource yet, if but " invalid " flag set, just show it is pre-allocation resource, so position and the quantity of pre-allocation resource in the blocks of files section of ext4 all is unfixed.This system can not be applied in file layouts such as ext3 and not adopt in the file system of piece segmented mode.

At above problem, the present invention has realized the resource allocation methods of a kind of ext2 of being applicable to, ext3 and ext4 system, thereby has not been subjected to file layout whether to adopt the restriction of piece segmented mode by adopting lasting predistribution technology.Below in conjunction with the drawings and specific embodiments the present invention is done detailed description.

Hit situation according to the pre-allocation of resources of history, determine the granularity blk_len of pre-allocation of resources, the length of the physical resource that representative is obtained is unit with the piece, corresponding to ee_len in the ext4 system, the two difference is that blk_len is not provided with " invalid " zone bit.Except determining the blk_len that by modes such as the user are provided with preferably determine in the following manner: the granularity of pre-allocation of resources is provided with a maximal value and a minimum value.If pre-allocation resource for the first time, it is minimum value that the predistribution granularity then is set; Otherwise, if last time, preallocated resource was hit in the actual allocated process, then adjust the pre-allocation of resources granularity and be predistribution last time granularity 2 times, and granularity adjustment maximum is no more than the maximal value of pre-allocation of resources granularity; If last time, preallocated resource was not hit in the actual allocated process, the pre-allocation of resources of then resetting granularity is the minimum value of pre-allocation of resources granularity.

After determining the pre-allocation of resources granularity, to the resource of the corresponding granularity of resource management parts application, obtain (blk_start, blk_len).Wherein blk_start is the starting block number of predistribution physical resource, 48 pieces that are spliced corresponding to ee_start_hi and ee_start in the ext4 system number.

In the index node of file, increase (blk_offset, blk_start, blk_len) territory, in this territory, write down (blk_offset lastingly, blk_start, blk_len), it is that the physics continuous resource of blk_len is pre-assigned to from file logical place blk_offset and begins the file logic continuous space that length is blk_len that expression begins length from physical location blk_start, and wherein file expectation logical place blk_offset is pre-allocation resource expection corresponding file logical resource starting position.

This document is inserted in the pre-allocation resource file chained list (pre_res_list), and there is pre-allocation resource at present in sign this document, in asynchronous removal process, can discharge this lasting pre-allocation resource.

When carrying out resources allocation, at first obtain and want Resources allocation file destination logical place file_offset for file.

If this document does not have pre-allocation resource at present, then carry out pre-allocation of resources, obtain (blk_start, blk_len).After predistribution finishes, at first piece blk_start is distributed to the logical place file_offset of file, and these relations of distribution are recorded in the file layout; Then, (blk_offset, blk_start blk_len) are that (file_offset+1, blk_start+1 blk_len-1) preserve lastingly, and the predistribution hit in tag in the file is set to revise the predistribution territory; At last, if blk_len-1 equals 0, then this document is deleted from pre-allocation resource file chained list, there has not been pre-allocation resource in expression this document.

If the current pre-allocation resource that exists of this document then detects the same pre-allocation resource of file_offset (blk_offset, blk_start, blk_len) relation of the blk_offset in.If file_offset equals blk_offset, expression predistribution is hit, and the predistribution hit in tag in the file then is set, and give the file_offset position blk_start resources allocation, revise lasting pre-allocation resource for (file_offset+1, blk_start+1, blk_len-1); If file_offset is not equal to blk_offset, expression predistribution is not hit, then remove the predistribution hit in tag in the file, and pre-allocation resource transferred to the file_offset place from blk_offset, give the file_offset position blk_start resources allocation, revise pre-allocation resource and be (file_offset+1, blk_start+1, blk_len-1).More than two kinds of situations, when blk_len-1 equals 0, this document is deleted from pre-allocation resource file chained list, the expression this document do not had pre-allocation resource.

Pre-allocation resource is fixed with the mapping relations of file logical place among the ext4, and does not allow migration, though the assurance of the continuity of file is very strong, may cause the pre-allocation resource quantity that keeps in the file too much, causes the resource excess waste; And the mapping relations among the present invention can dynamic transfer, when file_offset is unequal with blk_offset, when representing promptly that also predistribution is not hit, the expection logical place blk offset of pre-allocation resource from file, transfer to file destination logical place file_offset, pre-allocation resource corresponding file logical place is unfixing, has embodied dirigibility.Though the present invention under Random assignment file resource situation, can destroy the continuous dispensing mode of file, avoided the resource excess waste problem of ext4 file system, and the situation of Random assignment file resource is considerably less relatively, the influence that causes is very little.

When deleted file, resource in the releasing document layout; If (blk_len) middle blk_len is greater than 0 for blk_offset, blk_start for pre-allocation resource, there is pre-allocation resource in expression, also need by the resource management parts (blk_start, blk_len) resource discharges, and this document is removed from pre-allocation resource file chained list.

The file pre-allocation resource is carried out asynchronous recovery, need at first judge whether to need to trigger asynchronous collection process.This judges needs to consider two factors: the resource situation of entire system after the overall resource situation of system is lower than certain thresholding, needs to trigger this process to discharge resource; What of pre-allocation resource quantity, more when this quantity, surpass certain thresholding after, need to trigger this process, carry out the release of resource.As long as more than two conditions satisfy one, just need to trigger this asynchronous process.

In asynchronous removal process, need at first to determine how many resources are this removal process need discharge.Asynchronous recovery generally can once all not reclaimed all pre-allocation resources, is that some resource should not reclaim on the one hand, is that then asynchronous process is bigger to the influence of total system if once all reclaim in addition on the one hand.Adopt the mode once reclaim certain ratio, determine the resource quantity that removal process need discharge.

Secondly, the pre-allocation resource that needs to determine which file needs to discharge, and influence factor has two, and one is the time of file resource assigns access, the preferential file that did not for a long time all carry out resources allocation that discharges; Another is the quantity of the pre-allocation resource held of file, and preferential release held the more file of resource quantity.More than two factors need take all factors into consideration, pay the utmost attention to first factor.

At last, from pre-allocation resource file chained list, obtain the file that needs to discharge pre-allocation resource, resource is discharged by resource management module, this document is removed from pre-allocation resource file chained list.

Need to discharge because the present invention adopts the mode of chained list to write down which resource, only need predistribution idling-resource to the fixed position to reclaim and get final product,, improved the recall precision of the asynchronous release of resource so successfully avoided the traversal queries of whole file system.

The reclaim mechanism of pre-allocation of resources is not provided in the ext4 file system, and the space waste under the huge situation of quantity of documents is serious, and the present invention has proposed the reclaim mechanism of pre-allocation of resources in conjunction with assigning process, has successfully solved this problem.

To those skilled in the art, the present invention can adopt a kind of resource allocation system to realize, as shown in Figure 2, this system comprises following several sections:

The resource management parts 2 that are connected with storage resources 1; The control assembly 3 that is connected with above-mentioned resource management parts 2; Described control assembly 3 has the interface of the file 4 that receives pending needs assurance resource continuity distribution; Below each several part is elaborated:

The storage resources 1 that resource management parts 2 are used to safeguard carries out the operation such as distribution, release, inquiry of storage resources 1.Storage resources 1 is the linear address space with a constant volume, and this linear address space is to be better than the adjusting the distance visit of address space far away of the access performance in neighbor address space.

Control assembly 3 is made up of lasting predistribution parts, file continuous dispensing parts, pre-allocation resource file linked list maintenance parts, the asynchronous recovery part of file pre-allocation resource.Lasting predistribution parts are used for determining the pre-allocation of resources granularity that the application pre-allocation resource carries out persistence from resource management parts 2, increase (blk_offset, blk_start, blk_len) territory in file inode; File continuous dispensing parts are used for resources allocation, and the physical resource that blk_start is represented is mapped to blk_offset, and this map record in file layout, neofield content more; Pre-allocation resource file linked list maintenance parts are used for the chained list that management maintenance takies the file of pre-allocation resource; The asynchronous recovery part of file pre-allocation resource is used for the pre-allocation resource that does not have to use is reclaimed, to avoid the wasting of resources.Described above comprehensive, by the continuous resource request for allocation of control assembly 3 reception files 4,, guarantee the continuity that file resource distributes by lasting predistribution operation and the operation of file continuous dispensing; By the recovery that pre-allocation resource is finished in the operation and the operation of asynchronous recovery pre-allocation resource of pre-allocation resource file chained list.

To those skilled in the art, each above-mentioned function can adopt the mode of the appropriate combination of hardware, software or hardware and software to realize.Should be noted that and understand, under the situation that does not break away from the desired the spirit and scope of the present invention of accompanying Claim, can make various modifications and improvement the present invention of foregoing detailed description.Therefore, the scope of claimed technical scheme is not subjected to the restriction of given any specific exemplary teachings.

Claims (9)

1. resource allocation methods comprises following steps:
1) determine the pre-allocation of resources granularity, the application pre-allocation resource obtains physical resource starting block number;
2) in file inode, increase the territory, the starting block of log file expectation logical place, described physical resource number and described pre-allocation of resources granularity in described territory;
3) for the file destination logical place carries out resources allocation, wherein comprise, the physical resource of described physical resource starting block number expression be mapped to described file destination logical place, and described map record in file layout, upgrade described territory content.
2. method according to claim 1 is characterized in that, comprises step 4) releasing document layout resource after the described step 3), if there is pre-allocation resource, then discharges pre-allocation resource.
3. method according to claim 1 is characterized in that, described step 2) comprise file destination is inserted in the pre-allocation resource file chained list; Also comprise the asynchronous recovery file of step 5) pre-allocation resource.
4. method according to claim 2 is characterized in that, described step 2) comprise file destination is inserted in the pre-allocation resource file chained list; Described step 4) comprises from pre-allocation resource file chained list deletes this document; Also comprise the asynchronous recovery file of step 5) pre-allocation resource.
5. according to claim 3 or 4 described methods, it is characterized in that described step 5) comprises following steps:
According to system resource condition and pre-allocation resource situation, determine whether to trigger described asynchronous recovery file pre-allocation resource;
Determine the resource quantity that described asynchronous collection process reclaims and reclaim which resource;
Described pre-allocation resource is discharged, described file destination is removed from pre-allocation resource file chained list.
6. method according to claim 1 is characterized in that, described step 1) comprises according to historical predistribution hit in tag determines the predistribution granularity; Equal file expectation logical place if described step 3) comprises described file destination logical place, hit in tag then is set, otherwise remove hit in tag.
7. resource allocation system comprises:
The resource management parts that are connected with storage resources; The control assembly that is connected with described resource management parts; Wherein said control assembly has the interface of receiving target file;
Wherein said resource management parts are used for the storage resources of maintenance system;
Wherein said control assembly is made up of lasting predistribution parts and file continuous dispensing parts, wherein, described lasting predistribution parts are used for determining the pre-allocation of resources granularity, from described resource management parts, apply for pre-allocation resource, in file inode, increase the starting block number of log file expectation logical place, physical resource and the territory of pre-allocation of resources granularity; Described file continuous dispensing parts are used for resources allocation, wherein comprise, and the physical resource of described physical resource starting block number expression is mapped to described file destination logical place, and described map record in file layout, upgrade described territory content.
8. system according to claim 7 is characterized in that, described control assembly also comprises pre-allocation resource file linked list maintenance parts, is used to safeguard the index node of the file that occupies pre-allocation resource.
9. system according to claim 7 is characterized in that, described control assembly is the asynchronous recovery part of include file pre-allocation resource also, is used to reclaim idle pre-allocation resource.
CNB2007101794319A 2007-12-13 2007-12-13 A kind of resource allocation methods and system CN100557571C (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CNB2007101794319A CN100557571C (en) 2007-12-13 2007-12-13 A kind of resource allocation methods and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CNB2007101794319A CN100557571C (en) 2007-12-13 2007-12-13 A kind of resource allocation methods and system

Publications (2)

Publication Number Publication Date
CN101183318A true CN101183318A (en) 2008-05-21
CN100557571C CN100557571C (en) 2009-11-04

Family

ID=39448606

Family Applications (1)

Application Number Title Priority Date Filing Date
CNB2007101794319A CN100557571C (en) 2007-12-13 2007-12-13 A kind of resource allocation methods and system

Country Status (1)

Country Link
CN (1) CN100557571C (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101765208A (en) * 2008-12-26 2010-06-30 华为技术有限公司 Method for distributing resources, network equipment and wireless system
CN102158408A (en) * 2011-04-01 2011-08-17 华为技术有限公司 Method for processing data stream and device thereof

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101765208A (en) * 2008-12-26 2010-06-30 华为技术有限公司 Method for distributing resources, network equipment and wireless system
CN101765208B (en) * 2008-12-26 2013-11-06 华为技术有限公司 Method for distributing resources, network equipment and wireless system
CN102158408A (en) * 2011-04-01 2011-08-17 华为技术有限公司 Method for processing data stream and device thereof
CN102158408B (en) * 2011-04-01 2014-06-04 华为技术有限公司 Method for processing data stream and device thereof

Also Published As

Publication number Publication date
CN100557571C (en) 2009-11-04

Similar Documents

Publication Publication Date Title
US20160283124A1 (en) Multi-streamed solid state drive
JP5530012B2 (en) Method and system for storage address remapping for storage devices
KR101357397B1 (en) Method for tracking memory usages of a data processing system
CN103392167B (en) The caching method of distributed memory system, node
US9552165B2 (en) Memory allocation buffer for reduction of heap fragmentation
EP2645260B1 (en) File system hinting
TWI432959B (en) Storage subsystem and storage system architecture performing storage virtualization and method thereof
US20140068152A1 (en) Method and system for storage address re-mapping for a multi-bank memory device
US7814128B2 (en) Multi-volume file support
US6587915B1 (en) Flash memory having data blocks, spare blocks, a map block and a header block and a method for controlling the same
TWI428830B (en) Converting machines to virtual machines
US9575678B2 (en) Hierarchical allocation for file system storage device
US7930326B2 (en) Space allocation in a write anywhere file system
US8458425B2 (en) Computer program, apparatus, and method for managing data
US6912645B2 (en) Method and apparatus for archival data storage
US5644766A (en) System and method for managing a hierarchical storage system through improved data migration
CN101351772B (en) Method and system for sharing files among different virtual machine images
CN100470506C (en) Flash memory management based on sector access
US9690694B2 (en) Apparatus, system, and method for an address translation layer
Chang et al. An efficient management scheme for large-scale flash-memory storage systems
US9584312B2 (en) Methods and systems for storing and retrieving data
CN101334797B (en) Distributed file systems and its data block consistency managing method
US6745311B2 (en) Method of allocating clusters of computer readable medium to a file while minimizing fragmentation of the computer readable medium
US7818287B2 (en) Storage management system and method and program
US7949637B1 (en) Storage management for fine grained tiered storage with thin provisioning

Legal Events

Date Code Title Description
PB01 Publication
C06 Publication
SE01 Entry into force of request for substantive examination
C10 Entry into substantive examination
GR01 Patent grant
C14 Grant of patent or utility model