CN111221472B - Multi-block allocation strategy optimization method and system for disk space allocation - Google Patents
Multi-block allocation strategy optimization method and system for disk space allocation Download PDFInfo
- Publication number
- CN111221472B CN111221472B CN201911371401.7A CN201911371401A CN111221472B CN 111221472 B CN111221472 B CN 111221472B CN 201911371401 A CN201911371401 A CN 201911371401A CN 111221472 B CN111221472 B CN 111221472B
- Authority
- CN
- China
- Prior art keywords
- space
- disk
- allocation
- file
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a multi-block allocation strategy optimization method for disk space allocation, which comprises the steps of inquiring disk free space resources, judging whether to open a pre-allocation strategy according to an inquiry result, and adjusting the pre-allocation space by adopting a step adjustment mode under the mode of opening the pre-allocation strategy, so as to complete file data writing at least according to the available space condition of a disk; the method fully considers the disk space, the allocation request type and the available space in the system, satisfies the allocation of the continuous space of the large file as far as possible, reserves the disk fragmented space for the subsequent small file request, and is superior to the traditional multi-block allocation strategy of the Ext4 file system in the aspects of disk read-write performance and the stability of fragment quantity; the application also provides a system for implementing the method.
Description
Technical Field
The application relates to the technical field of disk optimization, in particular to a method and a system for optimizing a multi-block allocation strategy for disk space allocation.
Background
Data is permeated into each industry and business function field nowadays, and becomes an important production factor, and magnetic disks are used as important carriers for storing the data, so that the magnetic disks are extremely important in front of an information system which is gradually huge and complex in each industry unit nowadays; it is well known that the first (bare device) of disk partition creation must be formatted, i.e., a file system installed, through which a user can manage files and data within the disk; the file system organizes and allocates disk space, is responsible for file storage and protecting and retrieving stored files, and is a collection of all software and data related to managing files. The file system can conveniently manage all file data in the disk, provide access control and operation methods for users, and provide uniform and concise interfaces for the users at the same time, thereby facilitating the users to use various hardware resources.
Nowadays, the capacity of the hard disk is becoming larger and larger, whether it is a mechanical hard disk or a solid state hard disk, which means that more and more files are carried in the disk, and the difficulty of disk management is also higher and higher. Linux exists in tens of file system types, and different file systems manage disk space by adopting different methods, and each file system has advantages and disadvantages. Most Linux release versions currently use the fourth generation extended file system (Fourth extended filesystem, ext 4) as default and are upgrades to the third generation extended file system (Third extended filesystem, ext 3). Ext4 supports a large file system, adds metadata and log checksums to improve data reliability, improves resistance to disk fragmentation, and supports online defragmentation.
However, the Ext4 file system is deployed on a disk or a partition by default, and considering the limitation of the adjustability of the space capacity of the disk or the partition and the diversity of file data, the design of policies such as multi-block allocation, pre-allocation and the like is more focused on the universality, and the adoption of a fixed mode (supporting opening and closing and parameterization setting) does not support the automatic selection and parameter adjustment of allocation policies. Therefore, if the file system is applied to a disk device with dynamically adjustable space, the following disadvantages are caused:
firstly, when disk resources are tense, a preallocation strategy may reduce the read-write performance of the disk; for example, the pre-allocation space policy of the small file of the Ext4 file system is set in the superblock of the file system, where the size of the pre-allocation space is a fixed value, and when the disk resource is tense, the situation that the pre-allocation space is released and the large file can only be forced to use the fragmented space may frequently occur.
Secondly, intermittently generating a fragment file in the disk; for example, the disk fragments are particularly easy to generate in a mixed writing scene of large and small files, when fragments in the disk are more, the Ext4 file system can allocate discrete disk space for the subsequently written files until the continuity of file data is ensured again through a multi-block allocation and pre-allocation strategy after the space of the disk device is expanded.
In a word, the related parameters of the policies such as multi-block allocation, pre-allocation and the like of the Ext4 file system need to be manually set by a user according to the self situation, but cannot be automatically adjusted according to the use situation of the disk space in the subsequent business processing process, and the flexibility is lacking.
In view of this, the present application has been made.
Disclosure of Invention
Aiming at the defects in the prior art, the application provides a multi-block allocation strategy optimization method and system for disk space allocation.
In order to achieve the above purpose, the technical scheme of the application is as follows:
a multi-block allocation strategy optimization method for disk space allocation comprises the following steps of
Inquiring the idle space resources of the disk and judging whether to open the pre-allocation strategy according to the inquiring result;
under the mode that the pre-allocation strategy is opened, a step adjustment mode is adopted to adjust the pre-allocation space, wherein when the ratio of small files to large files of a disk processing request is in a first range and the available space of disk idle space resources is a first percentage, the allocation length of the group pre-allocation space is a first value; when the ratio of small files to large files of the disk processing request is in the first range and the available space of the disk free space resources is a second percentage larger than the first percentage, the allocation length of the group pre-allocation space is a second value larger than the first value;
and writing file data at least according to the available space condition of the disk.
Further, in the above method for optimizing a multi-block allocation policy for allocating disk space, the step adjustment mode further includes: when the ratio of small files to large files of the disk processing request is in a first range and the available space of the disk free space resources is a third percentage greater than the second percentage, the creation of the file pre-space is performed for the write request of the large file.
Further, the method for optimizing the multi-block allocation policy of disk space allocation includes
1) Under the scene that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk, the preallocation strategy is closed, wherein the tense is that the available space is less than 10%;
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported, expanding the disk space first and then retrying to process the file writing request; wherein the margin is that the available space is more than 50%;
3) And in the pre-allocation strategy opening mode, according to the calculation of the request length, adopting a step adjustment mode to adjust the pre-allocation space.
Further, the method for optimizing the multi-block allocation policy of disk space allocation includes, before selecting to allocate fragmented disk space for a file write request or attempting to expand disk space and then allocate continuous space for the disk space according to the available space condition of the disk, the scalability of the disk space and the file space allocation request length:
initializing a space allocation request;
the space available within the disk is scanned.
Further, in the above method for optimizing a multi-block allocation policy for allocating disk space, according to the available space condition of a disk, the scalability of the disk space, and the length of a file space allocation request, selecting to allocate fragmented disk space for a file write request or to try to expand the disk space before allocating a continuous space for the disk space, including:
when the disk supports an online capacity expansion function, if the space outside the disk is rich and allows the capacity expansion of the disk space, if the length of a query result is less than 20% of the requested length and the available space in the disk is less than 20% when a file space allocation request is processed, after the capacity expansion of the disk space is attempted, the available space in the disk is scanned again and allocated to the file for writing data;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, executing the initialization space allocation request for multiple times, scanning the available space in the disk and allocating fragmented disk space for the file writing request;
if the available space outside the disk supports the expansion of the disk but the space resources are not abundant, the disk expansion is performed when the query result length is less than 5% of the request length and the available space inside the disk is less than 5%.
The application also provides a multi-block allocation strategy optimization system for disk space allocation, which comprises a processor and a memory, wherein the memory stores a program, and when the program is run by the processor, the program is executed:
inquiring the idle space resources of the disk and judging whether to open the pre-allocation strategy according to the inquiring result;
under the mode that the pre-allocation strategy is opened, a step adjustment mode is adopted to adjust the pre-allocation space, wherein when the ratio of small files to large files of a disk processing request is in a first range and the available space of disk idle space resources is a first percentage, the allocation length of the group pre-allocation space is a first value; when the ratio of small files to large files of the disk processing request is in the first range and the available space of the disk free space resources is a second percentage larger than the first percentage, the allocation length of the group pre-allocation space is a second value larger than the first value;
and writing file data at least according to the available space condition of the disk.
Further, in the above-mentioned multi-block allocation policy optimization system for disk space allocation, the step adjustment mode further includes: when the ratio of small files to large files of the disk processing request is in a first range and the available space of the disk free space resources is a third percentage greater than the second percentage, the creation of the file pre-space is performed for the write request of the large file.
Further, in the above-mentioned multi-block allocation policy optimization system for disk space allocation, performing disk free space resource query includes
1) Under the scene that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk, the preallocation strategy is closed, wherein the tense is that the available space is less than 10%;
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported, expanding the disk space first and then retrying to process the file writing request; wherein the margin is that the available space is more than 50%;
3) And in the pre-allocation strategy opening mode, according to the calculation of the request length, adopting a step adjustment mode to adjust the pre-allocation space.
Further, in the above-mentioned multi-block allocation policy optimization system for disk space allocation, the program execution "set multi-block allocation flow adjustment mechanism, complete file data writing" includes
Initializing a space allocation request;
scanning available space in the disk;
according to the available space condition of the disk, the expandability of the disk space and the length of the file space allocation request, selecting to allocate fragmented disk space for the file writing request or to try to expand the disk space and then allocate continuous space for the disk space.
Further, in the above-mentioned multi-block allocation policy optimization system for allocating disk space, according to the available space condition of the disk, the scalability of the disk space, and the length of the file space allocation request, selecting to allocate fragmented disk space for the file write request or to try to expand the disk space before allocating continuous space for the disk space, including:
when the disk supports an online capacity expansion function, if the space outside the disk is rich and allows the capacity expansion of the disk space, if the length of a query result is less than 20% of the requested length and the available space in the disk is less than 20% when a file space allocation request is processed, after the capacity expansion of the disk space is attempted, the available space in the disk is scanned again and allocated to the file for writing data;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, executing the initialization space allocation request for multiple times, scanning the available space in the disk and allocating fragmented disk space for the file writing request;
if the available space outside the disk supports the expansion of the disk but the space resources are not abundant, the disk expansion is performed when the query result length is less than 5% of the request length and the available space inside the disk is less than 5%.
Compared with the prior art, the application has the beneficial effects that:
the multi-block allocation strategy optimization method for disk space allocation fully considers the disk space, allocation request types and available space in a system, satisfies the allocation of continuous space of large files as much as possible, reserves disk fragmented space for the subsequent small file requests, and is superior to the multi-block allocation strategy of the traditional Ext4 file system in the aspects of disk read-write performance and the stability of fragment quantity; the application also provides a system for implementing the method.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. Like elements or portions are generally identified by like reference numerals throughout the several figures. In the drawings, elements or portions thereof are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of stages of a prior art pre-allocation strategy implemented within a disk space allocation operational flow;
FIGS. 2A and 2B are schematic diagrams illustrating the prior art pre-allocation competing with file write requests for free disk space
FIG. 3 is a schematic diagram of a pre-allocation policy optimization scheme implementation according to one embodiment of the present application;
FIG. 4 is a diagram of the implementation of a multi-block allocation strategy optimization scheme according to one embodiment of the present application;
FIG. 5 is a schematic diagram illustrating the allocation and execution of multiple blocks of space in a disk-supported capacity expansion scenario according to an embodiment of the present application;
FIG. 6 is a logical block diagram of a multi-block allocation policy optimization system for disk space allocation in accordance with the present application.
Detailed Description
Embodiments of the technical scheme of the present application will be described in detail below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present application, and thus are merely examples, and are not intended to limit the scope of the present application.
It is noted that unless otherwise indicated, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs.
Example 1
The Ext4 file system has different pre-allocation policies for space allocation requests for size files. For space request of small file, a called group pre-allocation space (percpu local group) allocation method is adopted, wherein the size of the pre-allocation space is a fixed value (2 Mbytes) set in the super block and is irrelevant to the size of the request allocation space; all subsequent requests for allocation of small files by the CPU share the pre-allocated space in order to ensure that the small files are clustered together. For the space request of a large file, a file pre-allocation space (perinode prealloc) allocation method is adopted, wherein the pre-allocation space is related to the size of the space to be allocated and the size of the current file (the value is taken upwards according to the power of 2); the subsequent received space request preferentially inquires whether the pre-allocation space of the file maintained in the memory is available or not, and the method is used for solving the fragmentation problem under the concurrent allocation condition.
As shown in fig. 1, when a file system receives a file write request in a pre-allocation mode, adjustment of a space request length is first performed (stage 101), a pre-allocation space is calculated according to the request length (as described above, the pre-allocation space adopts a fixed calculation model) and the allocation request is adjusted; then, the available space in the disk is scanned (102 stage), and notice that the available space in the disk does not comprise the pre-allocated space maintained in the disk at the moment; after the multi-round scanning, if the idle space meeting the requirement is inquired, executing a stage 103, creating a relevant pre-allocation space for the current request and allocating a file writing request space to finish data writing, otherwise executing a stage 104, releasing all the pre-allocation spaces in the disk, returning to continue executing a stage 102 if the pre-allocation space is released, and returning to error if no pre-allocation space is released.
If the free space resources in the disk are tense, the stage 104 frequently occurs under the condition that the service pressure ratio of the file write request received by the file system is larger, and the pre-allocation space created in the stage 103 is released by other file write requests after not being used, so that the utilization rate of the pre-allocation space is reduced, and the disk fragments are increased. In addition, the space inquiry allocation length is larger than the file writing request length in the pre-allocation space mode, and the increase of the continuous space allocation length can increase the time consumption of the disk idle space scanning and reduce the service execution efficiency. Pre-allocation competes with file write requests for disk free space as shown in FIGS. 2A and 2B.
As shown in fig. 2A, processing a large file a write request, and sequentially executing the 101 phase-102 phase-103 phase in fig. 1, namely completing a file write task in a normal flow; the writing request of the small file B is processed, the length of the allocation request is over multiple times as the pre-allocation space strategy, and when the free space resources of the disk are tense, the stages 101, 102, 104, 102 and 103 in the figure 1 are sequentially executed, so that the release of the pre-allocation space of the disk is involved. Because of the amplification of the allocation request length, after the pre-allocation space is released (stage 104), the space meeting the writing request length of the small file B can be allocated only by one query, and the position shown in fig. 2A can be found by multiple queries, so that the time consumption of the query is increased, and the task time of the writing request of the file is prolonged.
As shown in fig. 2B, processing a large file N write request will sequentially execute the steps 101, 102, 103, 101, 102, 104, 102, 103 in fig. 1, where the large file N is split into two segments due to occupation of free pre-allocation space in the disk. If the preallocated space mode is closed when the writing request of the small file B is executed due to the shortage of the free space resources of the disk, the writing request of the large file N can avoid fragmentation, and the writing request can be completed only by executing the stage 101, the stage 102 and the stage 103.
To solve the above problem, optimizing the Ext4 file system pre-allocation policy herein involves implementing switch pre-allocation and automated adjustment of pre-allocation space length parameters, as shown in fig. 3.
Compared with the original strategy, the method has the advantages that the disk idle space resource query (stage 301) is added as reference information for subsequent processing, the space request interval and the length (stage 302) adopt a stepwise adjustment strategy, and the post-processing (stages 304 and 305) of the available space is related to the disk idle space resource condition, and the specific adjustment is as follows:
as shown in FIG. 3, a method for optimizing a multi-block allocation policy for disk space allocation includes
S1, establishing a pre-allocation strategy and an optimization mechanism, and allocating space for a file write request;
s2, setting a multi-block distribution flow adjustment mechanism to complete writing of file data.
The method supports parameterized starting and closing of the pre-allocation mode through the pre-allocation strategy, avoids the creation of invalid pre-allocation space and improves task processing time; the immobilized value of the pre-allocation space is replaced by adopting a mode of stepwise adjusting the pre-allocation space in a scene, so that the utilization rate of the pre-allocation space is improved; and increasing an online capacity expansion attempting mechanism by utilizing a multi-block allocation strategy, and reducing the file fragmentation degree under the condition of rich capacity expandable space of the disk so as to stabilize the read-write performance of the disk.
Specifically, step S1, a pre-allocation strategy and an optimization mechanism are established
The step optimizes the pre-allocation strategy of the Ext4 file system, and realizes the automatic adjustment of the switch pre-allocation and the pre-allocation space length parameters. Comprising the following steps:
s11, after receiving a file writing request, inquiring the idle space resources of the disk, and judging whether to open a pre-allocation strategy according to the inquiring result;
s12, adjusting the space through a stepwise adjustment strategy under the mode of starting a pre-allocation strategy;
s13, scanning available space in the disk, creating relevant pre-allocation space for the request and allocating file writing request space to complete data writing.
In step S11, the file system receives a file writing request, firstly, the file system performs inquiry on the free space resources of the disk, and confirms that the space state of the disk is in a rich, general or tension state and is used as reference information for subsequent processing; and scanning available space in the disk by combining the disk idle space resource query result and the adjustment result of the stepwise adjustment strategy, and performing pre-allocation and completing a write request:
1) Under the condition that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk (tense = less than 10 percent of available space), the preallocation strategy is closed, so that the extra inquiry time consumption caused by the increase of the space allocation request length of the preallocation space is avoided, and the disk data fragments are seriously generated due to the frequent creation and release of the preallocation space.
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported (the surplus=available space is more than 50%), the stage of releasing the preallocated space inside the disk is skipped when the file writing request is processed, the disk space is expanded first and then the file writing request is retried, so that the continuity of file data can be ensured.
In general, if the pre-allocation policy is opened when the free space resources of the disk are in tension, that is, when the service pressure ratio of the file write request received by the file system is greater due to the tension of the free space resources in the disk, it is required to frequently release all the pre-allocation spaces in the disk, which will cause that the created pre-allocation space is not used and is released by other file write requests, thereby resulting in a decrease in the utilization rate of the pre-allocation space and an increase in disk fragments. In addition, the space inquiry allocation length is larger than the file writing request length in the pre-allocation space mode, and the increase of the continuous space allocation length can increase the time consumption of the disk idle space scanning and reduce the service execution efficiency. Therefore, in the method, whether the pre-allocation strategy is opened or not is determined by inquiring the idle resources of the disk space, and the creation of the pre-allocation space is supported.
3) In the pre-allocation policy opening mode, calculating a pre-allocation space according to the request length (as described above, the pre-allocation space adopts a fixed calculation model) to adjust the request, wherein the pre-allocation space adjusting mode is related to disk free space resources and disk file types, and the step adjusting mode is adopted in the application:
the small files of the disk processing request are most (more than 70 percent), the allocation length of the pre-allocation space of the group is [ 2M=10% -20% of the available space, 4M=20% -30% of the available space, 8M=30% -50% of the available space, 32M=more than 50% of the available space ], and the file pre-allocation is executed for the large file writing request only when the available space exceeds 50%;
the large file is most (more than 70 percent) requested by disk processing, the allocation length of the group pre-allocation space takes the value of [256 K=10% -20% of available space, 512 K=20% -30% of available space, 1 M=30% -50% of available space, 2 M=more than 50% of available space ], and the file pre-allocation is canceled for the large file writing request when the available space is less than 10%;
the disk processing request is a file mixture with a size, the allocation length of the pre-allocation space of the group is [512 K=10% -20% of the available space, 1 M=20% -30% of the available space, 2 M=30% -50% of the available space, 4 M=more than 50% of the available space ], and when the available space exceeds 50%, the file pre-allocation space is adjusted to be rounded up to the power of 2, otherwise, the additional pre-allocation space is only 50% of the allocation request and does not exceed 1M.
In this embodiment, in the file writing request, the file size refers to the size of the disk space occupied by the file; files within 1MB, for example, are small files, and files greater than 1MB can be counted as large files.
In this way, the continuity of the file under the condition of space abundance is ensured; under the condition of space shortage, the number of large space requests is reduced (the pre-allocation can increase the space request length), and unnecessary time consumption is reduced; meanwhile, the automatic adjustment of the pre-allocation parameters can effectively improve the utilization rate of the pre-allocation space, reduce invalid allocation, and reduce disk fragments to improve the file read-write performance.
Calculating a pre-allocation space through a stepwise adjustment strategy, selecting an optimal result from the available space for multiple rounds of scanning, creating the pre-allocation space, and allocating a file write request space to finish data writing.
S2, setting a multi-block distribution flow adjustment mechanism to finish writing file data
The logical block mapping is replaced by an extension tree (section tree) in the Ext4 file system, and a plurality of data blocks can be mapped by one extension structure, so that the use of metadata blocks is reduced; the block distributor supports continuous multi-data block distribution, namely, a plurality of data blocks can be distributed by calling once according to the principle of distributing more continuous space as much as possible, so that the mechanism reduces the calling of the block distributor, improves the continuity of file data, reduces disk fragmentation and improves the performance of a system; the Ext4 file system mainly relates to disk partition, so that the processing of the allocation request by the multi-block allocation strategy is not fully considered under the condition that the disk space supports online capacity expansion; in the application, by setting a multi-block distribution flow adjustment mechanism, the multi-block distribution strategy is transplanted to the magnetic disk equipment or other data storage media supporting online capacity expansion, the read-write performance of the system is ensured, and the following optimization scheme is provided.
As shown in fig. 5, specifically includes:
s21, initializing a space allocation request;
s22, scanning available space in a magnetic disk;
s23, according to the available space condition of the disk, the expandability of the disk space and the length of the file space allocation request, selecting to allocate fragmented disk space for the file write request or to try to expand the disk space and then allocate continuous space for the disk space.
And (3) a plurality of allocation strategies designed by the Ext4 file system are used for preferentially allocating the existing space in the disk once the disk space is under tension, and the expansion of the disk space is not attempted until the disk space is insufficient. In this way, in the case where the disk space is tight and the fragmentation of the remaining free space is serious, if a large file write request is received, the data space allocated for the large file in the existing available space of the disk first cannot be guaranteed continuously and possibly is severely fragmented, as in the case of processing the large file a write request in fig. 4. This also means that if the disk supports online capacity expansion, the rate of fragments in the disk will increase stepwise (suddenly increase when disk space is tight and slow down after disk space expansion), making disk performance extremely unstable.
Therefore, in the multi-block allocation flow adjustment mechanism set in the above step 2 of the present application, s23, according to the available space condition of the disk, the scalability of the disk space, and the file space allocation request length, selects to allocate fragmented disk space for the file write request or first try to expand the disk space and then allocate continuous space for the disk space, includes:
when the disk supports an online capacity expansion function, if the disk space is allowed to be expanded by the margin of the space outside the disk, if the length of a query result is less than 20% of the length of a request and the available space in the disk is less than 20% when a file space allocation request is processed, after the disk space is attempted to be expanded (stage 504), the available space in the disk is scanned again to be allocated (stage 503) for writing data into the file;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, the allocation request is completed in the steps of 501, 502 and 503 for a plurality of times, and file data continuity is not ensured, namely, the step 504 is not executed any more; if there is space available outside the disk to support expansion of the disk but space resources are not abundant, the scene of the triggering stage 504 is more severe, and the expansion of the disk is performed only when the length of the query result is less than 5% of the request length and the available space in the disk is less than 5% (stage 504).
The method fully considers the disk space, the allocation request type and the available space in the system, satisfies the allocation of the continuous space of the large file as much as possible, reserves the disk fragmentation space for the subsequent small file request, is superior to the existing multi-block allocation strategy of the Ext4 file system in the aspects of the disk read-write performance and the stability of the fragment quantity, and is an ideal choice for supporting the online capacity expansion function of the disk.
Example 2
In another aspect, as shown in fig. 6, the present application further provides a system for implementing the method, including a processor and a memory, where the memory stores a program, and when the program is executed by the processor, the program performs:
establishing a pre-allocation strategy and an optimization mechanism, and allocating space for a file write request;
setting a multi-block distribution flow adjustment mechanism to complete the writing of file data.
Specifically, the program executes the steps of establishing a pre-allocation strategy and an optimization mechanism, allocating space for a file write request, including
After receiving the file writing request, inquiring the idle space resource of the disk, and judging whether to open the pre-allocation strategy according to the inquiring result;
under the mode of starting a pre-allocation strategy, adjusting the space through a stepwise adjustment strategy;
and scanning available space in the disk, creating a relevant pre-allocation space for the current request and allocating a file writing request space to finish data writing.
The Ext4 file system has different pre-allocation policies for space allocation requests of size files:
for the space request of the small file, a group pre-allocation space (percpu local group) allocation method is adopted, wherein the size of the pre-allocation space is a fixed value (2 Mbytes) set in the super block and is irrelevant to the size of the request allocation space; all subsequent requests for allocation of small files by the CPU share the pre-allocated space in order to ensure that the small files are clustered together. For the space request of a large file, a file pre-allocation space (perinode prealloc) allocation method is adopted, wherein the pre-allocation space is related to the size of the space to be allocated and the current file size (the value is taken upwards according to the power of 2); the subsequent received space request preferentially inquires whether the pre-allocation space of the file maintained in the memory is available or not, and the method is used for solving the fragmentation problem under the concurrent allocation condition.
In the application, a file system receives a file writing request, firstly, the file system performs inquiry on the free space resources of a disk, and confirms that the space state of the disk is in a rich, general or tension state and is used as reference information for subsequent processing; and scanning available space in the disk by combining the disk idle space resource query result and the adjustment result of the stepwise adjustment strategy, and performing pre-allocation and completing a write request:
1) Under the condition that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk (tense = less than 10 percent of available space), the preallocation strategy is closed, so that the extra inquiry time consumption caused by the increase of the space allocation request length of the preallocation space is avoided, and the disk data fragments are seriously generated due to the frequent creation and release of the preallocation space.
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported (the surplus=available space is more than 50%), the stage of releasing the preallocated space inside the disk is skipped when the file writing request is processed, the disk space is expanded first and then the file writing request is retried, so that the continuity of file data can be ensured.
3) In the pre-allocation policy opening mode, calculating a pre-allocation space according to the request length (as described above, the pre-allocation space adopts a fixed calculation model) to adjust the request, wherein the pre-allocation space adjusting mode is related to disk free space resources and disk file types, and the step adjusting mode is adopted in the application:
the small files of the disk processing request are most (more than 70 percent), the allocation length of the pre-allocation space of the group is [ 2M=10% -20% of the available space, 4M=20% -30% of the available space, 8M=30% -50% of the available space, 32M=more than 50% of the available space ], and the file pre-allocation is executed for the large file writing request only when the available space exceeds 50%;
the large file is most (more than 70 percent) requested by disk processing, the allocation length of the group pre-allocation space takes the value of [256 K=10% -20% of available space, 512 K=20% -30% of available space, 1 M=30% -50% of available space, 2 M=more than 50% of available space ], and the file pre-allocation is canceled for the large file writing request when the available space is less than 10%;
the disk processing request is a file mixture with a size, the allocation length of the pre-allocation space of the group is [512 K=10% -20% of the available space, 1 M=20% -30% of the available space, 2 M=30% -50% of the available space, 4 M=more than 50% of the available space ], and when the available space exceeds 50%, the file pre-allocation space is adjusted to be rounded up to the power of 2, otherwise, the additional pre-allocation space is only 50% of the allocation request and does not exceed 1M.
In this way, the continuity of the file under the condition of space abundance is ensured; under the condition of space shortage, the number of large space requests is reduced (the pre-allocation can increase the space request length), and unnecessary time consumption is reduced; meanwhile, the automatic adjustment of the pre-allocation parameters can effectively improve the utilization rate of the pre-allocation space, reduce invalid allocation, and reduce disk fragments to improve the file read-write performance.
Calculating a pre-allocation space through a stepwise adjustment strategy, selecting an optimal result from the available space for multiple rounds of scanning, creating the pre-allocation space, allocating a file write request space and completing data writing
Program execution "set Multi-Block Allocation flow adjustment mechanism, complete File data writing", including
Initializing a space allocation request;
scanning available space in the disk;
according to the available space condition of the disk, the expandability of the disk space and the length of the file space allocation request, selecting to allocate fragmented disk space for the file writing request or to try to expand the disk space and then allocate continuous space for the disk space.
The application selects to allocate fragmented disk space for a file writing request or try to expand the disk space and then allocate continuous space for the disk space according to the available space condition of the disk, the expandability of the disk space and the length of the file space allocation request by a set multi-block allocation flow adjustment mechanism, comprising the following steps:
when the disk supports an online capacity expansion function, if the disk space is allowed to be expanded by the margin of the space outside the disk, if the length of a query result is less than 20% of the length of a request and the available space in the disk is less than 20% when a file space allocation request is processed, after the disk space is attempted to be expanded (stage 504), the available space in the disk is scanned again to be allocated (stage 503) for writing data into the file;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, the allocation request is completed in the steps of 501, 502 and 503 for a plurality of times, and file data continuity is not ensured, namely, the step 504 is not executed any more; if the available space outside the disk supports the expansion of the disk but the space resources are not abundant, the scene of the triggering stage 504 is more severe, and when the length of the required query result is less than 5% of the request length and the available space inside the disk is less than 5%.
The system of the present application is used to implement the method of the present application described above, and the principle of implementation is described with reference to example 1 above.
Aiming at a storage system supporting online capacity expansion of a disk space, the system of the application improves the stability of the read-write performance of the disk and reduces the file data fragment rate when the disk space is tense by an optimization mechanism of a pre-allocation strategy and an excessive block allocation flow adjustment mechanism.
Firstly, in terms of the stability of the read-write performance of a disk, the application automatically sets the adjustment parameters of the pre-allocation space and opens and closes the conditional opening of the pre-allocation space, thereby ensuring that the pre-allocation space can be maximally utilized when the disk processes the read-write requests of multiple types of users (whether large and small files are read-written or massive small files or large files are read-written) during space allocation, and simultaneously avoiding the additional query expense caused by the pre-allocation space when resources are tensed and the allocation waste caused by frequent release of the pre-allocation space.
And secondly, when the disk space allocation executes the multi-block allocation and pre-allocation strategy, the application fully considers the characteristics of the read-write request (the file proportion of the size), the available space and the expandable state of the disk, the satisfaction of the query result on the request and the like, automatically adjusts the key strategy parameters, fully exerts the advantages of the allocation strategy, avoids the adverse effect caused by excessive strategies, particularly solves the problem of serious file fragmentation caused by excessive space pre-allocation when the idle resources of the disk are tense, and improves the flexibility of the multi-block allocation and pre-allocation strategy in the operation process.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application, and are intended to be included within the scope of the appended claims and description.
Claims (8)
1. A method for optimizing a multi-block allocation strategy for disk space allocation is characterized by comprising the following steps of
Inquiring the idle space resources of the disk and judging whether to open a pre-allocation strategy according to the inquiring result;
under the mode that the pre-allocation strategy is opened, a step adjustment mode is adopted to adjust the pre-allocation space, wherein when the ratio of small files to large files of a disk processing request is in a first range and the available space of disk idle space resources is a first percentage, the allocation length of the group pre-allocation space is a first value; when the ratio of small files to large files of the disk processing request is in the first range and the available space of the disk free space resources is a second percentage larger than the first percentage, the allocation length of the group pre-allocation space is a second value larger than the first value;
writing file data is completed at least according to the available space condition of the disk,
wherein, the method for inquiring the idle space resource of the disk comprises the following steps of
1) Under the scene that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk, the preallocation strategy is closed, wherein the tense is that the available space is less than 10%;
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported, expanding the disk space first and then retrying to process the file writing request; wherein the margin is that the available space is more than 50%;
3) And in the pre-allocation strategy opening mode, according to the calculation of the request length, adopting a step adjustment mode to adjust the pre-allocation space.
2. The method of optimizing a multi-block allocation policy for disk space allocation according to claim 1, wherein said step adjustment mode further comprises: when the ratio of small files to large files of the disk processing request is in a first range and the available space of the disk free space resources is a third percentage greater than the second percentage, creating a file pre-space is performed for the write request of the large file.
3. The method of optimizing a multi-block allocation policy for allocation of disk space according to claim 1, wherein before selecting to allocate fragmented disk space for a file write request or attempting to expand disk space before allocating contiguous space for it based on disk space availability, disk space scalability, and file space allocation request length, comprising:
initializing a space allocation request;
the space available within the disk is scanned.
4. The method for optimizing a multi-block allocation policy for allocating disk space according to claim 3, wherein selecting to allocate fragmented disk space for a file write request or to try to expand disk space before allocating contiguous space for the file write request according to the available disk space, the scalability of disk space, and the file space allocation request length comprises:
when the disk supports an online capacity expansion function, if the space outside the disk is rich and allows the capacity expansion of the disk space, if the length of a query result is less than 20% of the requested length and the available space in the disk is less than 20% when a file space allocation request is processed, after the capacity expansion of the disk space is attempted, the available space in the disk is scanned again and allocated to the file for writing data;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, executing the initialization space allocation request for multiple times, scanning the available space in the disk and allocating fragmented disk space for the file writing request;
if the available space outside the disk supports the expansion of the disk but the space resources are not abundant, the disk expansion is performed when the query result length is less than 5% of the request length and the available space inside the disk is less than 5%.
5. A multi-block allocation policy optimization system for disk space allocation, comprising a processor and a memory, wherein the memory stores a program that when executed by the processor performs:
inquiring the idle space resources of the disk and judging whether to open a pre-allocation strategy according to the inquiring result;
under the mode that the pre-allocation strategy is opened, a step adjustment mode is adopted to adjust the pre-allocation space, wherein when the ratio of small files to large files of a disk processing request is in a first range and the available space of disk idle space resources is a first percentage, the allocation length of the group pre-allocation space is a first value; when the ratio of small files to large files of the disk processing request is in the first range and the available space of the disk free space resources is a second percentage larger than the first percentage, the allocation length of the group pre-allocation space is a second value larger than the first value;
writing file data is completed at least according to the available space condition of the disk,
wherein, the method for inquiring the idle space resource of the disk comprises the following steps of
1) Under the scene that the free space resources of the disk are tense and the free space outside the disk is tense or does not support the online capacity expansion of the disk, the preallocation strategy is closed, wherein the tense is that the available space is less than 10%;
2) After inquiry, the disk free space resources are not strained, the preallocation strategy is opened, and the creation of the group preallocation space and the file preallocation space is supported; if the free space resources of the disk are insufficient, but the free space outside the disk is abundant and the online capacity expansion of the disk is supported, expanding the disk space first and then retrying to process the file writing request; wherein the margin is that the available space is more than 50%;
3) And in the pre-allocation strategy opening mode, according to the calculation of the request length, adopting a step adjustment mode to adjust the pre-allocation space.
6. The multiple block allocation policy optimization system of claim 5, wherein said ladder adjustment mode further comprises: when the ratio of small files to large files of the disk processing request is in a first range and the available space of the disk free space resources is a third percentage greater than the second percentage, creating a file pre-space is performed for the write request of the large file.
7. The system of claim 6, wherein the program executing before selecting to allocate fragmented disk space for a file write request or to try to expand disk space and then allocate contiguous space for the file write request based on disk space availability, disk space scalability, and file space allocation request length comprises
Initializing a space allocation request;
the space available within the disk is scanned.
8. The multi-block allocation policy optimization system of claim 7, wherein selecting to allocate fragmented disk space for a file write request or to try to expand disk space before allocating contiguous space for it based on disk space availability, disk space scalability, and file space allocation request length comprises:
when the disk supports an online capacity expansion function, if the space outside the disk is rich and allows the capacity expansion of the disk space, if the length of a query result is less than 20% of the requested length and the available space in the disk is less than 20% when a file space allocation request is processed, after the capacity expansion of the disk space is attempted, the available space in the disk is scanned again and allocated to the file for writing data;
if the space outside the disk is tense and does not allow the disk to be expanded arbitrarily, executing the initialization space allocation request for multiple times, scanning the available space in the disk and allocating fragmented disk space for the file writing request;
if the available space outside the disk supports the expansion of the disk but the space resources are not abundant, the disk expansion is performed when the query result length is less than 5% of the request length and the available space inside the disk is less than 5%.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371401.7A CN111221472B (en) | 2019-12-26 | 2019-12-26 | Multi-block allocation strategy optimization method and system for disk space allocation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371401.7A CN111221472B (en) | 2019-12-26 | 2019-12-26 | Multi-block allocation strategy optimization method and system for disk space allocation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111221472A CN111221472A (en) | 2020-06-02 |
| CN111221472B true CN111221472B (en) | 2023-08-25 |
Family
ID=70830842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911371401.7A Active CN111221472B (en) | 2019-12-26 | 2019-12-26 | Multi-block allocation strategy optimization method and system for disk space allocation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111221472B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101556557A (en) * | 2009-05-14 | 2009-10-14 | 浙江大学 | Object file organization method based on object storage device |
| CN102915340A (en) * | 2012-02-29 | 2013-02-06 | 浙江工商大学 | Expanded B+ tree-based object file system |
| CN103500089A (en) * | 2013-09-18 | 2014-01-08 | 北京航空航天大学 | Small file storage system suitable for Mapreduce calculation model |
| CN103577470A (en) * | 2012-08-03 | 2014-02-12 | 上海交通大学 | File system and method for improving performances of web server |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6978283B1 (en) * | 2001-12-21 | 2005-12-20 | Network Appliance, Inc. | File system defragmentation technique via write allocation |
-
2019
- 2019-12-26 CN CN201911371401.7A patent/CN111221472B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101556557A (en) * | 2009-05-14 | 2009-10-14 | 浙江大学 | Object file organization method based on object storage device |
| CN102915340A (en) * | 2012-02-29 | 2013-02-06 | 浙江工商大学 | Expanded B+ tree-based object file system |
| CN103577470A (en) * | 2012-08-03 | 2014-02-12 | 上海交通大学 | File system and method for improving performances of web server |
| CN103500089A (en) * | 2013-09-18 | 2014-01-08 | 北京航空航天大学 | Small file storage system suitable for Mapreduce calculation model |
Non-Patent Citations (1)
| Title |
|---|
| 栾德杰 ; 倪宏 ; 潘梁 ; 吴丽彬 ; 孙鹏 ; .一种基于启发式算法的数据块扩展预分配策略.微计算机信息.2010,(第35期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111221472A (en) | 2020-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104899156B (en) | A kind of diagram data storage and querying method towards extensive social networks | |
| US7506003B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| US20070016621A1 (en) | Moving data from file on storage volume to alternate location to free space | |
| CN110727685B (en) | Data compression method, equipment and storage medium based on Cassandra database | |
| TW200805134A (en) | Data operations in flash memories utilizing direct data file storage | |
| CN101373445B (en) | Method and apparatus for scheduling memory | |
| US20070016631A1 (en) | Moving data from file on storage volume to alternate location to free space | |
| US20200311026A1 (en) | File processing method and server | |
| CN111522507B (en) | Low-delay file system address space management method, system and medium | |
| CN107888687B (en) | Proxy client storage acceleration method and system based on distributed storage system | |
| CN111475507B (en) | Key value data indexing method for workload adaptive single-layer LSMT | |
| CN107181773B (en) | Data storage and data management method and device of distributed storage system | |
| CN106874465A (en) | Method for efficiently managing cache based on data version | |
| CN103399915A (en) | Optimal reading method for index file of search engine | |
| CN111221472B (en) | Multi-block allocation strategy optimization method and system for disk space allocation | |
| CN102609508B (en) | High-speed access method of files in network storage | |
| US7505986B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| US7873681B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| CN102298527A (en) | Method and implementation system for upgrading CPE (customer premises equipment) versions in parallel | |
| CN116226128A (en) | OLAP engine cold-hot separation optimization method taking object storage as cold storage | |
| CN113192558A (en) | Reading and writing method for third-generation gene sequencing data and distributed file system | |
| CN113344548A (en) | Workflow generation method, device, equipment and storage medium | |
| CN110825660A (en) | HBase-based optimization method, system and equipment for Julient suspension | |
| US7383292B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| CN105224677A (en) | A kind of database operation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |